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Abstract 

The  current  Air  Force  logistics  operations  system  is  reactive  in  nature, 
meaning  that  after  the  aircraft  detects  a  part  failure,  the  maintenance  person  must  perfonn 
fault  isolation  procedures  and  then  steps  are  taken  to  repair  or  replace  the  faulty  item. 

This  may  or  may  not  include  ordering  of  a  replacement  item  from  base  supply.  The 
Autonomic  Logistics  System  (ALS)  concept  changes  this  reactive  process  into  a 
proactive  one  with  the  employment  of  technologies  such  as  prognostics  and  distributed 
information  network.  This  new  approach  to  the  logistics  process  shows  the  potential  for 
cost  savings,  increased  aircraft  operational  availability,  and  better  system  performance. 

The  ALS  basic  function  can  be  compared  to  the  human  body’s  nervous  system. 
The  body’s  nervous  system  performs  activities  automatically  or  without  constant  thought 
like  respiration  and  blood  circulation.  This  same  concept  can  be  applied  to  the  Air  Force 
logistics  system.  Certain  logistics  tasks  can  be  handled  automatically  or  autonomously. 
Ordering  parts  for  a  broken  system,  calling  the  right  maintenance  specialist  to  the  right 
aircraft  reporting  a  problem,  notifying  the  maintenance  control  center  that  a  certain 
aircraft  has  a  malfunctioning  system  and  will  not  be  available  for  the  next  sortie,  and 
other  possible  applications.  Since  the  ALS  can  handle  these  routine  tasks,  the 
maintenance  personnel  are  free  to  perfonn  more  important  tasks  such  as  maintaining  and 
repairing  the  jets. 

This  thesis  explores  the  impact  of  this  concept  on  the  aircraft  sortie  generation 
process  by  building  a  discrete  event  simulation  tool  to  allow  the  study  of  the  baseline 
existing  system  and  the  ALS. 
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SIMULATION  OF  AUTONOMIC  LOGISTICS  SYSTEM  (ALS)  SORTIE 


GENERATION 


I.  Introduction 

The  Air  Force  needs  tools  that  allow  analysis  and  evaluation  of  newly  emerging 
operational  concepts.  This  thesis  builds  such  a  tool  through  the  use  of  modeling  and 
simulation.  This  tool  will  help  decision  makers  perform  what-if  analyses  to  determine 
whether  these  concepts  will  provide  benefits  over  current  systems.  These  benefits  may 
include,  a  cost  savings,  increased  operational  availability,  and  better  system  performance. 

The  Joint  Strike  Fighter  (JSF)  aircraft  program  office  is  developing  a  newly 
emerging  operational  concept  called  the  Autonomous  Logistics  System  (ALS).  The 
program  office  is  in  the  system  development  and  demonstration  phase  of  the  program 
with  Lockheed  Martin  as  the  prime  contractor.  This  new  logistics  system  shows  the 
potential  for  great  savings  over  the  current  way  the  Air  Force  conducts  logistics 
operations  by  employing  emerging  technologies  such  as  prognostics  and  making  use  of  a 
distributed  information  network  to  accelerate  the  information  flow.  This  thesis  builds  a 
discrete  event  simulation  tool  of  the  current  sortie  generation  process  and  the  new  ALS  to 
allow  comparisons  to  be  made  between  two  systems;  the  current  maintenance  system  and 
the  ALS. 
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Background 

The  current  Air  Force  logistics  operations  system  is  reactive  in  nature,  meaning 
that  after  the  aircraft  detects  a  part  failure,  the  maintenance  person  performs  fault 
isolation  procedures  and  then  any  steps  to  repair  or  replace  the  faulty  item.  This  process 
is  usually  started  when  the  aircraft  detects  a  fault,  which  sets  a  flag  and  sends  a  notice  to 
the  pilot.  Depending  on  the  severity  of  the  fault  the  mission  can  continue  and  the 
information  is  saved  for  maintenance  debrief.  The  new  ALS  is  a  proactive  approach  to 
the  logistics  operations.  Meaning  that  when  the  same  scenario  takes  place,  a  number  of 
actions  will  automatically  kick-off.  They  include,  but  are  not  limited  to,  isolating  the 
fault,  taking  the  necessary  actions  to  work  around  the  problem  and  possibly  continue  the 
mission,  notification  of  ground  personnel,  ordering  the  equipment  and  right  personnel  to 
repair  the  aircraft,  and  if  necessary,  even  ordering  a  replacement  part  from  base  supply. 
This  new  process  will  make  the  jet  more  reliable  and  easier  to  maintain  and  will  also 
make  the  weapon  system  more  affordable. 

The  JSF  is  not  the  only  program  addressing  these  new  technologies;  other 
organizations  are  investigating  this  new  logistics  concept.  The  Air  Force  Research 
Laboratory  (AFRL)  is  investigating  whether  a  prognostics  and  health  management 
(PHM)  system,  the  heart  of  the  ALS,  can  have  a  have  significant  impact  to  the  current 
aircraft  fleet  i.e.  F-16,  F-15,  C-5,  C-17,  etc.  Also,  the  Army  and  Navy  are  placing  a 
Health  Usage  Monitoring  System  (HUMS)  on  their  helicopter  fleet. 
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Autonomic  Logistics 

To  understand  an  ALS,  consider  the  basic  functions  of  the  human  body’s  nervous 
system.  The  body’s  nervous  system  performs  activities  automatically  or  without  constant 
thought  like  respiration  and  blood  circulation.  This  same  concept  can  be  applied  to  the 
Air  Force  logistics  system.  Certain  logistics  tasks  can  be  handled  automatically  or 
autonomously.  Ordering  parts  for  a  broken  system,  calling  the  right  maintenance 
specialist  to  the  right  aircraft  reporting  a  problem,  notifying  the  maintenance  control 
center  that  a  certain  aircraft  has  a  malfunctioning  system  and  will  not  be  available  for  the 
next  sortie,  are  some  of  the  possible  applications.  Since  the  ALS  can  handle  these  routine 
tasks,  the  maintenance  personnel  are  free  to  perform  more  important  tasks  such  as 
maintaining  and  repairing  the  jets. 

The  heart  of  the  ALS  is  a  fully  functional  PHM.  The  PHM  can  detect  aircraft 
system  faults,  perfonn  on-board  diagnostics  and  fault  isolation,  and  delay  maintenance  if 
a  system  can  either  be  reconfigured  or  is  not  required  for  the  next  mission,  and  report  its 
status  and  findings.  The  other  key  system  for  the  ALS  is  the  Distributed  Information 
System  (DIS).  This  is  the  information  side  of  the  ALS.  It  makes  available  the  PHM  data 
to  all  the  appropriate  logistics  functions,  keeping  them  informed  of  status  and  making 
requests  for  parts,  manpower  and  equipment  if  the  situation  needs  these  items. 

Previous  work 

Rebulanan  (2000)  and  Malley  (2001)  conducted  initial  thesis  research  into  the 
ALS  and  PHM  systems  respectively.  Rebulanan  constructed  an  Autonomic  Logistics 


3 


simulation  called  ALSim  as  a  tool  to  allow  comparison  between  ALS  and  the  current 
maintenance  process.  His  research  showed  that  higher  aircraft  availability  could  be 
obtained  with  an  ALS.  Malley  (2001)  built  on  that  research  by  adding  more  detail  to  the 
PHM  capability  in  ALSim.  The  PHM  capability  utilized  inputs  from  notional  JSF 
sensors  and  employed  an  artificial  neural  network  to  predict  remaining  service  life.  Both 
programs  were  written  in  the  JAVA®  programming  language. 

Discrete  Event  Simulation  Model  Development 

The  research  conducted  in  this  thesis  will  look  more  in-depth  at  the  failure 
information  and  will  take  a  different  approach  to  the  model  building.  The  model  is  built 
in  Arena®,  a  discrete  event  simulation  software  package.  The  model  is  structured  to 
allow  easy  insertion  of  new  objects  that  are  deemed  appropriate  for  implementation.  The 
model  will  include  many  aspects  of  a  minimal,  yet  effective,  ALS,  assumed  present  at  the 
base  level.  This  will  allow  comparison  between  current  system  procedures  and  the 
envisioned  ALS  of  the  future.  Sensitivity  analysis  of  the  model  will  pinpoint  areas  of 
greatest  concern  and  potential.  The  simulation  tool  will  examine  an  Air  Expeditionary 
Force  (AEF)  scenario.  A  wing  working  in  an  AEF  scenario  needs  assurance  that  their 
aircraft  can  safely  and  effectively  perform  the  deployed  mission  without  a  great  deal  of 
last  minute  inspections  and  parts  swapping.  This  simulation  tool  can  help  ascertain  if  the 
deployment  will  function  smoothly  in  the  tightly  constrained  AEF  deployment  time 
block.  Once  the  units  have  deployed,  the  simulation  tool  can  help  ascertain  if  supply  and 
maintenance  will  be  able  to  plan  for:  1)  component  failures,  and  2)  when  and  where  the 
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components  should  be  delivered  thus  minimizing  man-hours,  cost  of  spares  and 
transportation,  and  reducing  the  on  hand  valuable  components. 

Aircraft  Sortie  Generation  Process 

Aircraft  maintenance  is  the  heart  of  flight  line  operations  and  it  of  course  takes  the 
most  time.  The  sortie  generation  process  has  been  the  same  for  many  years.  It  takes 
several  different  specially  trained  personnel  to  generate  an  aircraft  for  the  day’s  mission. 
The  process  is  cyclical  in  nature;  Figure  1  shows  the  typical  process. 


Figure  1.  Sortie  Generation  Process 
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The  starting  point  is  generally  considered  to  be  the  aircraft  landing.  After  moving 
to  the  parking  location  and  engine  shutdown,  post  flight  servicing  is  conducted,  while  the 
aircrew  conducts  their  debriefings  to  the  maintenance  crew.  Numerous  routine 
maintenance  functions  are  required  to  ready  the  jet  for  the  next  mission,  plus  any 
unscheduled  maintenance  derived  from  the  recorded  faults  collected  during  the  flight. 

The  aircraft  are  prepared  for  flight  by  the  ground  crews,  the  pilots  load  the  assigned 
mission,  take-off,  perform  the  mission,  and  land  to  the  complete  the  cycle. 

The  ALS  hopes  to  make  positive  impacts  on  this  process.  These  impacts  should 
come  via  more  infonnation  sharing,  better  information,  and  more  timely  information. 

The  box  in  Figure  1  indicates  where  the  ALS  will  have  the  greatest  impact.  This  will  also 
be  the  place  that  we  will  focus  our  modeling  efforts. 

Problem 

Air  Force  decision  makers  need  a  simulation  tool  to  study  the  effects  of  the 
emerging  Autonomic  Logistics  System  (ALS)  technologies  on  the  Air  Force’s  sortie 
generation  process.  This  discrete  event  simulation  model  will  facilitate  quick  turn  studies 
providing  concrete  measures  of  performance. 

Research  Objective 

The  model  will  be  developed  in  Arena®  with  extensive  use  of  dynamic  graphics 
to  allow  for  better  definition  of  required  objects  and  interactions.  The  modules  from 
Arena®  will  be  written  such  that  object-oriented  code  may  be  quickly  developed  from  the 
model. 
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Scope 


This  research  will  develop  a  discrete  event  simulation  model  to  emulate  the 
effects  of  a  baseline  sortie  generation  process  and  a  fully  operational  Autonomic 
Logistics  System  (ALS)  to  include  a  prognostic  and  health  management  (PHM)  system 
and  distributed  information  system  (DIS)  at  the  base  level.  This  model  will  lay  the 
groundwork  for  an  object-oriented  model  that  will  eventually  give  decision  makers  the 
ability  to  simulate  existing  and  emerging  aspects  of  the  Autonomic  Logistics  System 
(ALS). 

The  baseline  scenario  is  the  existing  organization  level  logistics  operations.  The 
model  is  structured  to  allow  for  easy  incorporation  of  additional  or  updated  objects  (such 
as  types  of  aircraft,  maintenance  equipment/process,  refueling  equipment/process, 
operating  locations,  etc.)  The  model  focuses  on  the  F-16  aircraft  and  the  radar  subsystem 
more  specifically  using  actual  data  from  the  F-16  radar  systems  that  are  flying  at  Hill 
AFB,  UT.  The  code  is  constructed  to  ensure  easy  reusability  for  other  aircraft  or 
subsystems  that  a  user  wishes  to  consider. 

Thesis  Organization 

This  thesis  is  organized  in  five  chapters.  The  second  chapter  contains  a  detailed 
literature  review  on  the  following  subjects;  Sortie  Generation,  Autonomic  Logistics, 
Health  Usage  Monitoring  System,  and  other  simulation  programs. 

Chapter  three  describes  the  methodology  that  went  into  development  of  the 
Arena®  sortie  generation  model  and  where  the  data  was  obtained.  Chapter  four  contains 
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the  results  from  the  simulations  while  the  fifth  chapter  includes  conclusions  and 
recommendations  for  future  research. 
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II.  Literature  Review 


Introduction 

This  literature  review  examines  the  research  that  defines  the  sortie  generation  and 
the  autonomic  logistics  processes.  Defining  these  processes  will  help  with  the  model 
development  discussed  in  chapter  3.  This  chapter  also  includes  discussions  of  the  two 
key  ALS  components  PHM  and  DIS,  another  PHM  system  (HUMS),  condition-based 
maintenance  (CBM),  and  other  simulation  packages  that  were  located  during  the 
literature  review. 

Sortie  Generation  Process 

As  discussed  in  Chapter  1  the  sortie  generation  process  involves  several  factors 
and  entities.  The  requirements  for  maintenance  and  the  sortie  generation  process  are 
found  in  AFI2 1-101  (2002:12),  “Aircraft  and  equipment  readiness  is  the  maintenance 
mission.  The  maintenance  function  ensures  aircraft  and  equipment  are  safe  serviceable, 
and  properly  configured  to  meet  mission  needs.”  Flight  line  maintenance  includes 
processes  to  inspect,  service,  and  maintain  aircraft  on  the  flight  line.  An  important  factor 
in  maintenance  is  knowing  the  status  of  the  aircraft  to  include  the  systems  and 
subsystems  that  make  up  such  an  aircraft.  Today’s  maintainers  rely  on  the  on-board 
diagnostics  to  indicate  a  problem  during  post  flight  servicing  and  inspection.  Only  after 
this  notification  can  they  then  undertake  the  fault  isolation  and  repair  process.  As 
discussed  in  the  next  section,  ALS  equipped  aircraft  will  conduct  a  majority  of  these 
tasks  autonomously  while  at  the  same  time  keeping  the  maintainers  infonned  of  weapon 
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system  status,  making  their  job  a  great  deal  easier  by  pointing  to  specific  parts  that 
require  replacement  or  indicating  when  such  maintenance  is  required.  As  indicated  in 
Chapter  1,  we  wish  to  identify  time  savings  due  to  an  ALS  system,  as  well  as  other 
benefits  from  such  a  system. 

Figure  1  defines  the  sortie  generation  process;  an  ALS  can  have  a  positive  impact 
throughout  the  process.  Starting  at  the  top  of  Figure  1,  the  landing  procedure,  an  ALS 
equipped  aircraft  can  relay  critical  system  information  while  the  aircraft  waits  for  the 
runway  to  become  available  and  after  landing  while  it  taxis  back  to  the  parking  location. 
Currently,  some  aircrews  report  the  aircraft  status  over  the  radio,  giving  the  ground  crews 
a  head  start  on  the  repair  process.  However,  this  status  usually  only  includes  the 
maintenance  fault  lists  (MFL)  flags.  After  engine  shutdown  the  aircrew  may  conduct  a 
short  discussion  with  the  ground  crew  before  heading  to  the  mission  debrief  area.  During 
parking  and  recovery,  the  aircraft  is  prepared  for  ground  operations,  and  aircraft  servicing 
commences.  This  servicing  includes  checking  fluid  levels  and  refueling  the  aircraft. 
During  the  aircrew  debriefing,  involving  the  aircrew  and  maintenance  personnel,  any 
discrepancies  are  discussed,  documented,  and  placed  into  a  computerized  information 
system,  in  the  F-16  case  the  Core  Automated  Maintenance  System  (CAMS).  The 
maintenance  personnel  are  looking  for  both  current  faults  and  repeat  faults.  Next,  if 
required,  a  maintenance  crew  heads  to  the  aircraft  to  conduct  the  repair  to  return  the 
aircraft  to  operational  status.  This  is  referred  to  as  unscheduled  maintenance  since  these 
faults  occur  over  the  course  of  the  sortie  mission,  meaning  maintenance  was  unplanned. 
Figure  2  shows  the  possible  steps  in  this  maintenance  process. 
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Figure  2.  Unscheduled  Maintenance  Process 


The  process  is  cyclical  with  steps  possibly  skipped  based  on  the  severity  and  type 
of  the  repair.  A  great  deal  of  time  is  saved  using  an  ALS  system.  Parts  are  ordered 
ahead,  since  the  PHM  system  has  already  diagnosed  and  isolated  the  fault. 
Documentation  is  created  by  the  PHM  and  “signed  off  by  the  maintainer  once  the  repair 
is  complete.  Lastly,  a  PHM  instills  confidence  that  the  correct  part  was  removed  and 
replaced.  The  last  step  on  many  fault  isolation  trees  includes  a  statement  that  reads, 


‘replace  item  A,  test  system,  if  fault  still  exists  replace  item  B.”  This  type  of  fault 
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isolation  can  be  frustrating  and  time  consuming.  However,  with  accurate  PHM,  the 
correct  item  is  removed  and  replaced  the  first  time. 

The  next  step  may  involve  preventative  maintenance  and  periodic  inspections, 
Time  Compliance  Technical  Order  installations,  system  calibrations,  and  Time  Change 
Item  (TCI)  replacements.  TCIs  are  conducted  on  the  critical  parts  replaced  or  repaired 
based  on  accumulated  flight  hours,  not  based  on  part  condition.  The  TCI  area  is  another 
area  where  PHM  and  condition  based  maintenance  may  supercede  the  need  for  these 
maintenance  actions.  The  PHM  can  check  system  status  and  indicate  when  parts  require 
maintenance,  rather  the  traditional  way  of  changing  parts  based  on  operating  hours. 

The  next  steps  prepare  the  aircraft  for  the  next  mission  and  may  include  weapon 
loading,  software  loading,  fuel  adjustment,  etc.  Now  the  aircraft  is  ready  for  preflight 
inspections.  The  crew  chief  conducts  his  inspection  and  then  the  aircrew  perfonns  their 
inspection.  The  aircraft  engine  is  started,  and  the  last  few  items  are  completed,  and  the 
aircraft  taxis  to  the  end  of  the  runway.  Certain  aircraft  require  an  end  of  runway 
inspection  prior  to  aircraft  launch.  Post-launch  cleanup  is  then  conducted;  storing  of  fire 
extinguishers,  inlet  covers,  etc.,  and  conducting  foreign  object  damage  (FOD)  inspections 
(Cardona  and  Sanford,  2001).  Figure  3  shows  a  notional  timeline  of  a  sortie  indicating 
how  an  ALS  can  save  time  in  the  repair  process. 


12 


Current  System 


ALS 


Order  part 


Figure  3.  Notional  Timeline  for  an  Aircraft  with  and  without  an  ALS  system 

In  the  current  process,  the  ground  crew  is  only  notified  of  a  fault  prior  to  landing 
if  the  pilot  radios  in  or  on  the  ground  after  engine  shutdown.  However,  this  is  usually 
just  the  maintenance  fault  lists  (MFL)  list  and  the  ground  crew  does  not  have  the 
diagnostics  to  determine  the  problem.  The  aircraft  is  diagnosed  to  determine  the  part  that 
needs  replaced.  The  part  is  ordered  only  after  this  process  is  complete. 

For  this  same  scenario  under  the  ALS,  the  onboard  PHM  would  notify  the  DIS 
through  a  Passive  Aircraft  Status  System  (PASS)  of  a  fault  and  which  part  to  order.  This 
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order  action  notifies  the  appropriate  aircraft  maintenance  specialist.  In  theory,  the  right 
maintained  the  right  part,  and  the  right  equipment  meet  the  aircraft  upon  its  return.  This 
significantly  reduces  the  time  to  repair  the  jet  and  return  it  to  operational  status.  The 
autonomic  logistic  concept  relies  on  two  main  objects  to  make  the  system  function.  First, 
is  the  PHM  system,  the  heart  of  the  ALS.  PHM  continuously  monitors  the  aircraft  for 
problems  to  include  pending  failures  as  well  as  after  the  fact  failures.  The  other 
important  factor  is  the  Distributed  Information  System  (DIS),  responsible  for  tracking 
and  relaying  the  maintenance  requirements  to  appropriate  elements  of  the  logistics  chain. 

Autonomic  Logistics 

According  to  Smith  et  al  (1999),  the  autonomic  logistics  concept  grew  out  the 
JSF  Concept  Exploration  phase  with  the  Advanced  Integrated  Diagnostics  Study.  “The 
study  effort  reviewed  current  aircraft  systems  and  available  technologies  for  promising 
techniques  in  prognostics,  diagnostics,  sensors,  diagnostic  design  tools,  maintenance 
systems  and  software  systems  (Smith,  1999:13).”  The  goal  is  to  move  logistics  from  the 
current  reactive  mode  to  a  proactive  one.  During  JSF  Concept  Development  the  concept 
architecture  was  developed  from  a  program  called  Advanced  Strike  Integrated 
Diagnostics  (ASID).  “Architecture  features  -such  as  the  diagnostic  design  process, 
benefits  of  integrated  information  flow  and  feedback  between  operation,  support  and 
design  function  -  were  found  to  be  important  attributes  (Smith,  1999:13).” 
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PHM 

According  to  Becker  et  al  (1998)  machine  prognostics  is,  “where  failure  modes 
and  the  remaining  life  of  a  system  can  be  predicted  (Becker,  1998:20).”  This  is  a  simple 
way  to  think  of  prognostics.  A  useful  analogy  is  the  way  doctors  make  prognosis  of 
human  health.  Diagnostics  tests  such  as  X-rays  and  blood  tests  are  conducted  and 
analyzed  by  lab  technicians  and  other  specialists  after  which  the  doctor  prescribes  a 
treatment  plan.  The  same  process  applies  to  electrical  and  mechanical  systems.  Sensors 
provide  reasoners  and  electronic  decision  makers  with  current  system  status  and 
operating  conditions  and  these  reasoners  make  a  prognosis  of  system  health.  If  a  problem 
is  pending,  a  notification  is  sent  to  the  flight  line  maintainer.  According  to  Becker  et  al, 
(1998),  “The  aim  of  prognostics  is  to  stop  disabling  of  fatal  failures  before  they  happen 
(Becker,  1998:20).” 

According  to  Ferrell  (2000)  the  requirements  for  a  PHM  are: 

a.  Provide  system  health  status  to  enable  pilot  alerts,  reconfiguration,  graceful 
degradation,  and  system  capability  assessment  in  the  event  of  failure. 

b.  Detect  and  isolate  failures  and  report  failure  information  to  trigger  the 
Autonomic  Logistics  process  to  effect  maintenance. 

c.  Collect  and  analyze  component  performance  data  to  predict  remaining  life  of 
selected  components  for  the  purpose  of  enhancing  safety  and  better  maintenance  and 
spares  planning. 
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DIS 

The  Distributed  Information  System  (DIS)  is  more  than  just  a  stream  of  aircraft 
data;  it  takes  the  aircraft  data  and  coverts  it  into  information.  The  following  is  a  proposed 
list  of  that  information: 

a.  Maintenance  Information/knowledge 

b.  Supply  chain  management  information 

c.  Health  and  usage  information 

d.  Forecast  aircraft  availability  data 

e.  Best  use  of  resource  recommendations 

f.  Training  Management  (Henley:  2000) 

In  addition  to  providing  the  above  information,  the  DIS  also  tracks  system, 
subsystem,  and  part  trends  and  other  issues.  This  additional  data  is  used  to  better  manage 
the  aircraft  fleet.  If  a  particular  part  is  showing  signs  of  degradation  and  that  aircraft  is 
scheduled  for  other  repairs,  then  the  degraded  part  is  replaced  along  with  the  scheduled 
maintenance. 

The  Passive  Aircraft  Status  System  (PASS)  is  a  component  of  the  DIS  that  speeds 
the  infonnation  transfer.  The  PASS  functions  autonomously,  first  storing  PHM  data  and 
then  downlinking  that  information  to  the  ground  units  of  the  logistics  chain.  AFRL 
conducted  a  study  of  existing  concepts  and  possible  implementation  of  these  for  use  on 
legacy  aircraft.  AFRL  was  able  to  devise  an  architecture  and  build  a  desktop 
demonstration  in  JAVA®.  The  demonstration  showed  possible  timing  of  data  transfer 
and  the  appearance  of  potential  computer  screens  for  the  maintainer  (Botello:  2000). 
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Other  PHM  Systems 

The  Army  and  Navy  are  currently  fielding  the  Health  and  Usage  Monitoring 
Systems  (HUMS)  into  their  helicopter  fleet.  This  was  an  important  factor  for  the 
condition  based  maintenance  (CBM)  policy  that  the  Navy  wants  to  implement  (Schaefer 
and  Haas:  2002;  Deaton  and  Glenn:  1999).  CBM  is  not  a  new  concept,  a  great  deal  of 
articles  have  been  published  for  ground-based  machinery.  Caterpillar  seems  to  lead  the 
field  in  this  implementation  with  on-board  diagnostics  and  remote  infonnation  passage 
back  to  a  central  processing  facility.  The  Navy  is  moving  towards  this  type  of 
maintenance  policy  and  away  from  the  standard  time-based  policy.  The  standard  time- 
based  policy  specifies  that  items  be  repaired  or  replaced  based  on  flight  hours  or  some 
other  usage  time  limit.  The  CBM  policy  is  to  ascertain  the  condition  of  the  critical  part 
through  sensor  outputs  and  reasoner  algorithms  to  repair  or  replace  the  parts  only  after  a 
certain  wear  is  detected. 

Schaefer  and  Haas  investigated  the  impact  of  HUMS  on  the  Navy’s  logistics 
processes  using  a  simulation  model  to  examine  HUMS  impact  on  process  reengineering, 
sensor  failures,  and  false  alann  rates  (Schaefer  and  Haas:2002).  They  focused  on  the 
operational  level  of  helicopter  logistics,  specifically  ten  high  maintenance  subsystems. 
The  processes  with  the  model  were  based  on  a  database  of  information  which  included, 
mean  time  to  repair  (MTTR),  unscheduled  and  scheduled  maintenance  man-hours  per 
flight  hour,  maintenance  action,  aircraft,  and  flight.  They  provide  some  interesting 
conclusions.  First,  they  found  that  the  baseline  phase  maintenance  or  scheduled 
maintenance  represented  a  bottleneck,  and  second  that  false  alann  rates  have  an  adverse 
impact  on  availability  (Schaefer  and  Haas:2002). 
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Another  program  the  Navy  is  investigating  is  called  the  Thinline  Health 
Monitoring  System  (THMS).  This  was  a  demonstration  applied  to  their  Submarine 
Towed  Array  Systems  (TAS).  The  TAS  is  a  critical  submarine  mission  component  that  if 
it  fails  to  operate,  repairs  are  deferred  until  the  submarine  returns  to  port  (Bishop  and 
Matzelevich:  2001:  5).  The  contractor,  Arete  Associates  and  Life  Cycle  Engineering 
(LCE)  constructed  this  proof-of  concept  device  to  operate  at  the  submarine  level,  similar 
to  aircraft  level  for  the  Air  Force.  The  THMS  device  included  data  collection,  storage, 
health  assessment,  and  a  prognostic  capability.  This  device  utilized  a  Bayesian  Belief 
Network  as  its  prognostics  engine.  The  contractors  showed  that  this  proof-of-concept  is 
feasible  and  that  it  has  the  potential  to  increase  the  TAS  operational  availability  and 
provide  a  cost  savings. 

Condition-Based  Maintenance 

There  are  numerous  papers  on  condition-based  maintenance  (CBM).  This  section 
discusses  a  few  related  to  Department  of  Defense  issues.  According  to  Nickerson  and 
Nemarich  (1990),  CBM  is  motivated  by  the  increasing  complexity  in  Naval  machinery 
and  the  decreases  in  the  military  budgets.  The  Navy,  like  the  Air  Force,  utilizes  a  time- 
based  repair  process  rather  than  evaluating  the  condition  of  the  machine  to  determine  if 
maintenance  is  necessary.  CBM  needs  several  items  to  operate.  First  of  all  monitoring 
systems,  algorithms  to  turn  data  into  system  state  infonnation,  and  forecasting  abilities  to 
predict  future  states  (Nickerson  and  Nemarich:  1990).  According  to  Nickerson  and 
Nemarich,  Table  1  shows  the  potential  benefits.  They  caution  that  all  tools  of  the  system 
must  be  in  place  to  receive  the  benefits. 
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Table  1.  CBM  Benefits  (Nickerson  and  Nemarich:  1990) 


Reduce  maintenance  induced  failures 

50% 

Reduce  maintenance  actions 

35% 

Increase  Availability 

20% 

Reduce  inspection  and  repair  hours 

20% 

Reduce  spare  parts  provisioning 

20% 

Reduce  good  parts  removal 

10% 

Extend  equipment  life/overhaul  cycle 

10% 

Several  companies  and  universities  have  formed  a  consortium  to  examine  ways  to 
incorporate  CBM.  This  consortium  is  called  Machinery  Infonnation  Management  Open 
Systems  Alliance  (MIMOSA).  The  member  organizations  include  Boeing,  Caterpillar, 
MIMOSA,  Newport  News  Shipbuilding,  Oceana  Sensor  Technologies,  Penn  State  ARL, 
Rockwell  Automation  and  Rockwell  Scientific.  According  to  Mitchell  (2002)  the 
alliance  has  the  “initial  objectives  of  providing  a  common,  open  protocol  for  exchanging 
complex  condition  information.”  The  consortium  is  striving  towards  a  JAVA®  based 
extensible  Markup  Language  (XML)  viewer  for  the  transfer  of  maintenance  information 
over  the  Internet. 

Other  Simulation  Projects 

Several  simulation  research  projects  have  been  conducted  in  the  area  of  sortie 
generation.  It  is  appropriate  to  highlight  those  related  to  this  research.  These  simulation 
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projects  come  from  academia,  small  disadvantaged  businesses,  larger  companies  and 
government.  Two  previous  AFIT  theses  directly  relate  our  research  effort.  These  were 
the  work  of  Capt  Rene  Rebulanan  and  Capt  Mike  Malley,  both  former  Graduate 
Operations  Research  (GOR)  students. 

Rebulanan  (2000)  developed  an  ALS  simulation  model  simulating  the  sortie 
generation  process  both  with  and  without  the  ALS.  The  purpose  of  his  research  was  to 
examine  system  performance  and  interaction  of  the  ALS  with  the  logistics  chain 
(Rebulanan:  2000;  5).  Rebulanan’s  model  was  written  in  the  JAVA®  program  language 
using  the  SILK®  software  package  and  was  called  ALSim.  Rebulanan  concentrated  on 
specifying  the  impact  of  the  ALS  on  the  sortie  generation  process.  The  main  parts  of  the 
model  included  the  Joint  Distributed  Information  System  (JDIS),  PHM,  and  the  logistics 
maintenance  and  supply  chain  (Rebulanan:  2000;  7).  The  main  measures  of  effectiveness 
of  this  model  were  aircraft  availability,  number  of  sorties  generated  and  time  waiting  for 
supply.  The  results  showed  that  a  statistically  significant  higher  availability  rate,  and 
number  of  sorties  generated  could  be  obtainable  with  an  ALS  installed  in  the  aircraft. 
However,  the  wait  time  was  not  significantly  reduced  when  employing  the  ALS 
(Rebulanan:  2000:  56). 

This  research  parallels  Rebulanan’s  work  establishing  a  more  generic  approach  to 
ALS  while  expanding  the  functionality  of  the  model  components.  In  addition,  this  effort 
incorporates  more  input  data  from  the  REMIS  system  and  other  emerging  historical 
databases  to  provide  a  closer  match  to  the  real  sortie  generation  system. 

Malley  (2001)  extended  Rebulanan’s  research  adding  more  definition  to  the  PHM 
system  in  the  ALSim  model  (Malley:  2001:  27).  Since  a  true  JSF  PHM  is  still  in 


20 


development  and  data  was  not  available  for  modeling  purposes,  Malley  generated 
notional  sensor  signals  used  by  the  PHM.  PHM  performance  was  based  on  Artificial 
Neural  Networks  (ANN)  trained  to  evaluate  when  a  part  was  degrading  past  a  pre- 
established  point  of  perfonnance  and  then  to  predict  this  point  based  on  assigned  failure 
thresholds.  These  failure  thresholds  could  be  modified  to  study  the  effects  of  the 
different  levels.  The  study  focused  on  the  probabilities  of  false  alarm  and  detection  and 
their  associated  times  (Malley,  2001:  28).  Malley’s  research  provided  a  detailed 
approach  to  model  the  JSF  PHM  and  some  associated  issues  that  may  be  encountered. 

Models  of  the  sortie  generation  process  have  also  been  developed  using 
Distributed  Interactive  Simulations.  These  are  large-scale  simulations,  which  typically 
involve  players  from  several  different  units  at  locations  spread  across  the  world.  They 
are  usually  run  in  real-time  with  no  single  simulation  in  control  of  the  entire  process 
(Miller:  Class  handouts,  OPER671).  These  simulations  are  becoming  increasingly 
important  due  to  the  fact  that  flying  hours  and  war  games  are  so  expensive.  These 
simulations  are  very  detailed  and  complex,  however  they  tend  to  leave  out  an  important 
aspect  of  the  simulation;  the  logistics  of  the  operations.  According  to  Banks  and  Styz 
(1997),  the  simulations  set  sortie  generation  rates  too  high  and  logistics  operations 
beyond  capabilities.  Banks  and  Styz  developed  a  tool  called  the  Airbase  Logistics 
System  in  AFIT’s  Virtual  Environments  Laboratory  as  a  means  of  “accurately  portraying 
airbase  logistics.”  Since  it  was  too  expensive  to  build  a  tool  from  scratch,  their  approach 
used  existing  tools  that  were  integrated  to  interface  with  current  Distributed  Interactive 
Simulations  systems. 
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The  government  and  its  contractors  have  built  other  stand-alone  simulation  tools 
to  model  AF  logistics  such  as  SIMFORCE  (2000),  LogSam™  (2002),  and  LCOM 
(1990).  The  largest  of  these  tools  is  the  LCOM  model  developed  by  the  RAND 
Corporation  for  the  Air  Force.  LCOM  has  been  in  use  since  1972  and  has  been  validated 
against  actual  data  including  Desert  Storm  data.  SIMFORCE  and  LogSam™  are 
relatively  new  software  tools  that  were  developed  by  Kelley  Logistics  Support  Services, 
and  Synergy  Corporation  respectively.  SIMFORCE  is  written  in  Arena®.  Below  is 
more  detail  on  each  model. 

LCOM 

The  Logistics  Composite  Model  (LCOM)  was  adopted  as  the  Air  Force  Standard 
for  modeling  logistics  manpower  requirements.  LCOM  is  a  large-scale  computer 
simulation  used  to  model  manpower  and  other  logistical  requirements  (XO  website: 
2002).  It  can  handle  large  and  small  weapon  systems  and  contains  data  preparation 
modules,  a  main  simulation  program,  post  processors,  and  models  Air  Force  direct 
maintenance  activities  at  the  wing  level.  LCOM  is  used  to  evaluate  weapons  system 
logistic  resource  impact  due  to  weapons  system  modification  and  is  used  for  what-if 
analyses  during  weapon  system  acquisition  (XO  web  site;  2002).  The  fundamental 
processes  modeled  are  requirements  for  sortie  demand,  system  reliability  and 
maintainability,  unscheduled  and  scheduled  maintenance  tasks,  and  ability  to  service 
demands  (people,  parts,  equipment).  LCOM  measures  of  effectiveness  include: 

•  Operations  (e.g.  sorties  flown,  missions  cancelled); 

•  Activities  (e.g.  average  time  to  complete,  resource  wait  time); 

•  Personnel  (e.g.  man-hours  utilized); 
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•  Supply  (e.g.  number  of  spares  backordered); 

•  Shop  repair  (e.g.  number  of  items  repaired); 

•  Equipment  (e.g.  equipment  used);  and 

•  Aircraft  (e.g.  number  of  aircraft  available)  (XO  web  site;  2002) 

LCOM  utilizes  historical  databases  CAMS,  and  Reliability  and  Maintainability 

Information  System  (REMIS)  along  with  the  engineering  data,  Logistics  Support 
Analysis  (LSA),  during  it’s  processing.  The  model  is  written  in  SimScript  II. 5  and  runs 
on  an  UNIX  HP750  or  personal  computer.  Various  verification  and  validation  audits 
include  a  favorable  comparison  to  Desert  Storm  data  (XO  web  site:  2002).  “LCOM 
stochastically  models  the  logistics  (personnel,  spare  parts,  test/repair  equipment/facilities) 
required  to  support  a  weapon  system  operation  under  a  given  scenario”  (XO  web  site: 
2002).  The  process  steps  modeled  are  shown  in  Figure  4. 


Figure  4.  LCOM  Sortie  Generation  Process 

Figure  4  shows  how  the  basic  process  flows,  but  not  shown  are  some  of  the  key 
details  of  the  simulation.  Stochastic  random  variables  are  used  to  detennine  first  if  a  part 
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is  broken,  then  also  if  a  replacement  part  is  available  for  installation,  and  are  the 
personnel  and  equipment  available  to  repair  the  aircraft. 

The  next  section  highlights  a  desktop  simulation  tool  built  by  a  small 
disadvantaged  business.  This  model  simulated  the  sortie  generation  process  but  their 
approach  was  more  focused  on  usability  by  the  maintenance  leadership.  These 
maintenance  managers  can  use  Scalable  Integration  Model  for  Objective  Resource 
Capability  Evaluations  (SIMFORCE)  at  their  desktop  and  do  not  require  a  long  set-up 
time  to  get  results. 


SIMFORCE 

The  Scalable  Integration  Model  for  Objective  Resource  Capability  Evaluations 
(SIMFORCE)  was  built  by  Kelley  Logistics  Support  Services  and  is  written  in  the 
Arena®  software  language.  SIMFORCE  simulates  the  wing  level  logistics  activities  to 
include  the  manpower,  equipment,  and  facilities  constraints.  It  was  built  to  allow 
decision  makers  to  formulate  what-if  problems  and  analyze  maintenance  manpower 
utilization  rates.  For  example,  a  decision  maker  can  determine  how  many  crew  chiefs 
would  be  required  if  a  certain  sortie  generation  rate  was  desired.  The  inputs  and  outputs 
are  devised  and  analyzed  in  Excel®  (Brown  and  Powers:  2000).  Although  LCOM  and 
SIMFORCE  simulate  the  same  process  their  intended  users  differ.  LCOM  provides  the 
Air  Force  leadership  with  manpower  estimates  for  new  weapons  systems  or  revised 
estimates  for  proposed  changes.  SIMFORCE  provides  wing  level  maintenance  mangers 
a  decision  support  tool  for  conducting  what-if  problems,  for  example,  “How  is  next 
week’s  deployment  going  to  affect  their  ability  to  conduct  operations?” 
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LogSAM™ 

The  Logistics  Simulation  and  Analysis  Model  (LogSAM™)  is  built  by  Synergy 
Inc.  LogSAM™  also  simulates  the  aircraft  sortie  generation  process.  The  model  is 
broken  down  into  several  modules,  aircraft  generation,  sortie  generation,  preflight  and 
launch,  and  post  flight  evaluation  (Smiley:  1997).  Added  features  include  its  ability  to 
schedule  sorties  based  on  the  Air  Tasking  Orders  (ATO).  These  ATOs  describe  what 
targets  to  attack  along  with  numbers  and  types  of  aircraft  to  use  (DTIC  web  site:  2002). 
Synergy  has  also  expanded  LogSAM™  to  include  a  module  called  LogBase™. 
LogBase™  simulates  enemy  attacks  and  the  effect  those  attacks  have  on  sortie  generation 
capability  (Synergy  web  site:  2002).  The  other  models  discussed  can  simulate  combat 
conditions  (increase  sortie  activity)  however,  it  seems  that  LogSAM™  has  the  unique 
feature  of  simulating  attacks  from  the  enemy. 

The  three  models  discussed  above  all  have  a  certain  unique  purpose.  LCOM  is  a 
large  government  operated  simulation  tool  that  provides  excellent  manpower  prediction. 
SIMFORCE  is  a  desktop/web  enabled  tool  allowing  the  maintenance  manager  to 
formulate  and  run  what  if  scenarios.  LogSam™  is  another  desktop  tool  that  not  only 
conducts  sortie  generation  simulations  but  allows  war-time  losses  to  be  simulated.  They 
all  simulate  the  sortie  generation  process,  however,  they  simulate  today’s  flying 
conditions  and  not  the  future  possibilities  that  will  exist.  This  research  will  attempt  to 
investigate  these  evolving  technologies  and  their  impact  on  the  aircraft  sortie  generation 
process. 
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Conclusion 


The  basic  sortie  generation  process  has  remained  constant  over  the  past  few  years. 
An  aircraft  flies  a  sortie,  lands,  is  parked  and  recovered,  is  serviced,  the  aircrew  debriefs 
the  maintenance  personnel,  the  aircraft  is  checked  for  failures,  if  none  exist,  it  is 
scheduled  and  then  prepped  for  the  next  mission,  taxis  out  and  takes  off  for  another 
sortie.  This  cyclical  process  is  repeated  according  to  the  daily  flying  schedule  or  until 
either  a  failure  occurs  or  phase  maintenance  is  required.  If  a  failure  occurs  the  aircraft  is 
sent  to  unscheduled  maintenance  and  several  other  actions  are  conducted  to  repair  the 
aircraft  is  the  most  expedite  manner. 

As  the  literature  search  indicates  several  simulation  tools  have  been  developed  to 
investigate  the  process,  as  it  exists  today.  However,  several  new  concepts  are  emerging 
to  make  the  process  more  efficient  to  save  limited  maintenance  resources.  The  ALS 
concept  is  one  of  these  concepts  that  are  being  investigated.  The  literature  search 
indicates  a  gap  in  tools  to  simulate  the  sortie  generation  process  along  with  the  affect  of 
ALS  on  this  process.  This  research  builds  a  simulation  tool  to  address  ALS  affects  on  the 
sortie  generation  process. 
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III.  Methodology 


Introduction 

This  chapter  describes  the  Autonomic  Logistics  Systems  sortie  generation 
simulation  model  built  for  this  research  effort.  The  following  sections  contain  the  key 
model  assumptions,  the  model  definition,  and  the  results  of  the  input  analysis  that  was 
conducted  on  three  data  sets.  The  model  was  built  to  represent  the  aircraft  sortie 
generation  process  defined  in  Figure  1  from  Chapter  1.  A  few  differences  from  Figure  1 
exist  due  to  the  nature  of  not  being  able  to  model  the  real  world  exactly  and  certain 
instances  where  changes  were  necessary. 

Assumptions 

Several  assumptions  were  made  during  the  model  building  process,  these  are 
broken  down  into  model  scope,  key  distinctions  from  the  real  process,  and  output 
differences  from  the  real  process. 

This  model  simulates  the  F-16  aircraft  sortie  generation  operations  but  is  scoped 
to  only  cover  in  detail  the  four  LRUs  that  make-up  the  AN/APG-68  radar.  The  other 
subsystems  are  modeled  to  experience  failures  at  preflight  inspection,  where  they  fail  per 
a  percentage  of  scheduled  sorties.  When  these  other  systems  fail,  troubleshooting, 
testing,  and  documentation  maintenance  tasks  are  carried  out  that  are  discussed  in  later 
sections  but  no  parts  are  removed  or  replaced  nor  supply  ordered.  The  entire  supply 
system  is  not  modeled  either,  the  delivery  of  parts  from  the  depot  to  the  flight  line  are 
simulated  via  delays  and  transfer  of  simulated  parts.  The  manpower  resources  are  built  to 
simulate  the  various  logistics  specialties  such  as  crewchiefs,  various  maintenance 
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specialists,  refuelers,  weapon  specialists,  etc.  but  their  entire  day  is  not  fully  modeled. 

For  ease  in  modeling,  these  manpower  resources  are  available  24  hours  a  day  so  that  they 
can  be  seized  by  the  aircraft  wherever  required.  This  means  that  these  resources  have 
low  utilization  since  they  are  not  required  for  work  while  the  aircraft  is  flying  or  when  it 
is  waiting  for  the  next  day’s  scheduled  takeoff.  The  model  was  run  with  eight  refuelers, 
eight  weapons  specialist,  sixteen  crewchiefs,  sixteen  maintenance  specialists  for 
debriefing,  and  sixteen  phase  inspection  resources  available  to  service  the  aircraft.  There 
were  four  maintenance  specialists  available  to  repair  the  aircraft.  These  numbers  are  not 
typical  base  manning  levels,  but  provide  a  reasonable  pool  of  resources  since  the  focus  of 
this  effort  was  to  measure  the  ability  to  produce  sorties  and  increase  aircraft  availability 
with  the  ALS  system  not  to  consider  the  impact  of  manpower  usage.  Other  sortie 
generation  models  discussed  in  Chapter  2  can  perform  these  kinds  of  studies. 

The  next  few  assumptions  are  where  the  model  does  not  replicate  the  real  process. 
First,  the  Can  not  Duplicate  (CND)  and  Retest  Okay  (RTOK)  events  were  not  accounted 
for  in  the  model.  These  happen  on  the  flight  line  or  at  the  test  bench  when  a  reported 
failure  cannot  be  repeated  or  the  bench  testing  shows  no  faults  to  be  repaired 
respectively.  However,  the  false  alarms  of  the  ALS  PHM  were  modeled  so  that  the 
analysis  could  include  these  actions.  A  false  alann  is  where  the  PHM  system  indicates  a 
failure  but  one  really  does  not  exist.  These  false  alarms  mimic  CND  and  RTOKs  since 
these  are  not  true  LRU  failures.  Also  as  will  be  discussed  later,  the  worst  case  impact  is 
assumed  for  false  alarms  with  part  removal  and  replacement  simulated  for  every 
occurrence.  If  an  aircraft  is  sent  to  unscheduled  maintenance  from  a  preflight  failure, 
only  troubleshooting,  operational  checkout,  and  documentation  are  conducted.  If  an 
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aircraft  is  flagged  for  a  LRU  failure  then  a  part  is  removed  and  replaced  every  time.  The 
diagnostics  gives  a  100%  accuracy,  which  is  not  realistic  in  the  real  world,  however, 
again  the  goal  of  this  research  was  to  compare  ALS  performance  to  current  day 
operations.  Partial  mission  capable  rates  are  not  calculated  since  there  is  only  the  one 
system  being  simulated.  The  aircraft  only  has  three  maintenance  states,  preflight  failure, 
radar  failure,  or  phase  maintenance. 

The  next  assumptions  note  how  the  output  differs  from  the  real  world.  Possessed 
aircraft  hours  equals  the  total  simulation  time.  This  is  true  since  no  aircraft  leave  flying 
status  (they  do  not  get  deployed  or  sent  to  the  depot).  The  Mission  Capable  rate  is 
calculated  by  subtracting  the  Not  Mission  Capable  rates  for  Supply  and  Maintenance. 

We  are  only  explicitly  modeling  a  single  aircraft  system  (radar  system)  and  how  the 
failures  of  this  system,  with  or  without  ALS,  affect  our  selected  MOEs.  Therefore, 
resource  levels  and  other  model  parameters  were  selected  to  obtain  a  reasonable 
(approximately  80%)  overall  Mission  Capable  rate  for  our  baseline  model. 

The  model  does  not  specifically  account  for  the  time  for  not  mission  capable  for 
both  supply  and  maintenance.  The  aircraft  is  either  waiting  on  supply  or  it  is  being 
maintained  (maintenance  personnel  always  available  with  modeled  resource  levels).  The 
simulation  is  modeled  to  simulate  24  hours  a  day  operations  five  days  a  week.  Typically 
each  resource  only  works  about  8  hours  a  day.  The  weekends  are  not  simulated  even 
though  they  may  be  used  in  the  real  world  to  repair  aircraft. 
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F-16  AN/APG-68  Radar 


“The  AN/APG-68  is  an  X-band,  all  weather,  multimode  Fire  Control  Radar 
(FCR)  featuring  extensive  Air-to-Air  (A/ A)  and  Air-to-Ground  (A/G)  capabilities” 
(Castrigno,  2002;  20).  It  consists  of  four  LRUs;  the  antenna,  the  modular  low  power 
radio  frequency,  the  dual  mode  transmitter,  and  the  advanced  programmable  signal 
processor  (Castrigno,  2002;  54).  These  LRUs  are  abbreviated  ANT,  MLPRF,  DMT,  and 
APSP  respectively  in  the  model.  A  portion  of  the  data  for  this  research  was  obtained 
from  the  Reliability  and  Maintainability  Information  System  (REMIS)  database 
maintained  by  Air  Force  Material  Command.  It  was  complied  over  a  two-year  period 
examining  F-16  failures  at  Hill  AFB,  UT.  The  details  of  how  this  data  was  analyzed  are 
found  in  the  input  analysis  section. 

Input  Data  Analysis 

The  input  data  was  drawn  from  several  sources.  The  following  explains  how  the 
data  were  transformed  for  use  in  the  simulation  model.  The  first  data  set  was  used  to 
determine  the  sortie  duration.  This  data  came  from  the  Data  Transfer  Cartridge  (DTC)  of 
the  aircraft  located  at  Hill  AFB,  UT.  The  DTC  data  was  from  a  three-month  period 
during  the  summer  of  2002.  Hill  AFB  takes  care  of  transferring  the  data  to  a  Microsoft 
Access  database.  From  there  the  data  was  transferred  to  Microsoft  Excel®  spreadsheets 
for  transforming  the  time  recording  into  raw  data.  Next,  the  data  was  put  through  the 
Arena®  Input  Analyzer  to  determine  an  appropriate  distribution  for  modeling  the 
process.  Evaluating  the  mean  square  error  of  the  curve  fit  between  the  empirical  data  and 
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the  selected  theoretical  distributions,  the  normal  distribution  looked  most  reasonable. 
Also,  after  plotting  the  data,  it  seemed  to  closely  match  the  nonnal  distribution. 

Another  data  set  from  Hill  AFB  contained  the  sortie  rates,  MC,  and  TNMCM 
rates  by  tail  number  for  the  388th  Fighter  Wing.  This  data  spanned  the  year  2001.  This 
data  was  used  mostly  to  double-check  the  output  of  the  simulation. 

The  third  data  set  was  18  months  of  REMIS  data  used  to  determine  the  LRU 
MBTF.  A  better  name  may  be  mean  time  between  LRU  replacement,  because  the 
analysis  did  not  include  reviewing  bench  level  and  depot  level  testing  to  determine  if  this 
LRU  was  a  CND  or  RTOK  unit.  The  analysis  that  was  done  included  scaling  the  REMIS 
database  down  to  just  the  tail  numbers  from  Hill  AFB  and  then  further  cutting  the 
database  back  to  only  the  tail  numbers  for  the  4th  Fighter  Group.  This  provided  enough 
data  for  analysis  to  determine  the  mean  time  between  replacement.  Lining  up  the 
operating  times  and  looking  at  the  difference  of  operating  time  between  LRU 
replacement,  and  then  averaging  these  numbers  across  the  twenty  tail  numbers,  the  mean 
time  between  replacement  was  calculated.  Table  4  shows  notional  MBTF  numbers.  The 
database  did  not  include  LRU  failures  for  ASPS  for  that  timeframe  so  twenty  other  tail 
numbers  were  selected  from  the  larger  database  to  calculate  the  mean  time  between 
replacement. 

Model  Introduction 

The  first  items  to  be  described  are  the  different  views  of  functional  areas  setup  in 
the  model.  Taking  advantage  of  the  named  views  in  Arena®,  several  of  these  views  were 
established  for  ease  in  navigation.  Table  2  shows  these  functional  areas  and  provides  the 
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associated  hot  key  used  to  recall  that  view.  This  will  also  be  the  template  for  the 
following  paragraphs  that  describe  the  individual  areas  of  the  sortie  generation  model. 
The  model  is  setup  with  Arena’s®  stations  and  routings  so  that  white  space  exists 
between  the  different  areas  that  were  simulated.  These  station  and  routing  modules  move 
the  aircraft  entities  between  the  appropriate  areas.  This  white  space  also  was  done  to 
make  the  model  easier  to  understand  and  made  changes  easier  as  the  model  evolved. 
Another  benefit  of  the  station  and  routing  concept  was  that  both  the  baseline  sortie 
generation  and  ALS  function  exist  in  one  model.  The  setting  of  one  variable  on  the 
graphical  user  interface  defines  whether  the  ALS  is  on  or  off. 


Table  2.  Model  Views 


View 

Hot  Key 

Animation 

(a) 

Create 

(c) 

Mission  Preparation 

(m) 

Preflight  Inspection 

(P) 

Aircraft  Launch 

(t) 

Flying 

(f) 

Landing 

(1) 

Parking  and  Recovery 

(r) 

Servicing  and  Debrief 

(s) 

Failure  Checking 

(d) 

Preventative  Maintenance 

(v) 

Hold 

(h) 

Unscheduled  Maintenance 

(u) 

Supply 

(y) 

PHM 

(z) 

This  model  is  a  collection  of  process  delays,  decision  modules,  and  routing 
stations  setup  to  simulate  the  baseline  sortie  generation  process  or  the  ALS  process 
depending  on  what  output  is  desired  by  the  user.  The  majority  of  process  delay  times  are 


from  several  subject  matters  experts  interviewed  during  the  course  of  the  model  building 
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and  a  RAND  study  on  sortie  generation  (2002:  3).  Table  3  shows  these  processes, 
associated  times,  and  cumulative  time. 


Table  3.  Model  Process  Times 


Area 

Mean  Process 

Times  (min) 

Cumulative 
Mean  Times 
(min) 

Turn  Around 
Times  (min) 

Turn  Around 
Cumulative  Mean 
Times  (min) 

Mission  Preparation 
Refuel 

10 

10 

5 

5 

Mission  Preparation 
Re-arm 

30 

40 

15 

20 

Preflight 

60 

100 

10 

30 

Engine  start,  Final 
system  checks,  Taxi, 
&  Arm 

10 

110 

10 

40 

Takeoff 

3 

113 

3 

43 

Fly 

61 

174 

61 

104 

Fand  and  Taxi 

15 

189 

15 

119 

Park  &  Recovery 

7 

196 

7 

126 

Servicing 

60 

256 

60 

186 

Debrief 

15 

271 

15 

201 

Preventative  and  Unscheduled  maintenance  and  supply  times  are  not  included  in 
the  table  since  not  every  aircraft  goes  through  those  processes  every  flight.  These  times 
are  described  in  their  associated  paragraphs  below. 


Graphical  User  Interface 

The  model  is  built  with  a  graphical  user  interface  (GUI)  to  allow  the  user  to 
change  any  of  the  twenty-two  different  parameters  prior  to  each  replication.  A  picture  of 
the  GUI  is  shown  in  Figure  5.  Table  4  includes  the  list  of  variables  and  attributes  that  can 
be  changed  and  a  brief  description  of  each.  These  values  are  the  baseline  starting  points 
for  each  of  the  variables.  The  false  alarm  and  PHM  Level  1  variables  will  be  changed 


33 


from  run  to  run  to  make  comparisons  from  the  existing  sortie  generation  operations  to 
one  with  an  ALS. 


Figure  5.  Graphical  User  Interface 
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Table  4.  GUI  Variable  Description 


Variable 

Description 

Initial 

Value 

Units 

attANTfail 

Time  until  failure  of  the  ANT  LRU 

375 

hours 

attAPSPfail 

Time  until  failure  of  the  APSP  LRU 

425 

hours 

attDMTfail 

Time  until  failure  of  the  DMT  LRU 

550 

hours 

attMLPRFfail 

Time  until  failure  of  the  MLPRF  LRU 

275 

hours 

varSupplyLevelANT 

Initial  supply  of  ANT  LRUs 

7 

N/A 

varSupplyLevelAPSP 

Initial  supply  of  APSP  LRUs 

7 

N/A 

varSupplyLevelDMT 

Initial  supply  of  DMT  LRUs 

7 

N/A 

varSupplyLevelMLPRF 

Initial  supply  of  MLPRF  LRUs 

7 

N/A 

varOrderLevelANT 

Order  level  for  the  ANT  LRU 

6 

N/A 

varOrderLevelAPSP 

Order  level  for  the  APSP  LRU 

6 

N/A 

varOrderLevelDMT 

Order  level  for  the  DMT  LRU 

6 

N/A 

varOrderLevelMLPRF 

Order  level  for  the  MLPRF  LRU 

6 

N/A 

varTakeoffl 

Takeoff  time  for  the  1st  group  of  4  A/C 

0800 

hours 

varTakeoff2 

Takeoff  time  for  the  2nd  group  of  4  A/C 

1000 

hours 

varTakeoffl 

Takeoff  time  for  the  3rd  group  of  4  A/C 

1200 

hours 

varTakeoff4 

Takeoff  time  for  the  4th  group  of  4  A/C 

1400 

hours 

varPreflightFail 

A/C  that  will  fail  the  preflight  inspection 

5 

percent 

varFalseAlarm 

A/C  that  will  experience  a  false  alarm 

3 

percent 

PHMLevel 

Level  for  aircraft  to  receive  maintenance 

10 

hours 

PHMBit 

Determines  if  PHM  if  on  or  off 
(0  =  off  /  1  =  on) 

1 

N/A 

varSecondPHMLevel 

Level  for  aircraft  to  wait  for  maintenance 
and  return  to  taxi  or  flying 

2 

hours 

NumTum 

Number  of  A/C  to  perform  a  turn  around 
flight 

2 

N/A 

Animation 

The  animation  area  contains  resource  modules,  queues,  stations,  and  routing  lines 
that  are  used  to  display  the  aircraft  and  supply  entities  and  the  paths  that  they  move  on 
throughout  the  simulation.  Also,  plots  are  generated  for  the  status  of  key  output 
parameters.  Figures  6  and  7  show  these  areas. 
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Figure  6.  Animation  Area 
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Figure  7.  Dynamic  Plots 
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Some  of  the  set-up  was  modeled  after  Arena®  examples  and  a  paper  from  the 
literature  search  that  showed  how  forming  zones  made  the  animation  better  (Raivio  et  al: 
2001).  The  animation  was  also  very  helpful  in  program  debugging.  Monitoring  how  the 
aircraft  and  supply  entities  progress  through  the  system  helped  to  make  sure  that  they  are 
moving  according  to  their  intended  flow. 

Create 

The  sixteen  aircraft  entities  are  created  in  the  model  at  the  start  of  each 
replication.  Figure  8  shows  a  view  of  this  area. 


Figure  8.  Aircraft  Create  Area 
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These  aircraft  entities  enter  into  the  sortie  generation  cycle  shown  in  Figure  1. 
These  entities  are  created  once  and  never  leave  the  simulation.  However,  the  entities  are 
not  running  continuously  in  the  model,  they  are  held  at  the  end  of  a  simulated  day, 
waiting  for  the  next  takeoff  cycle  to  begin.  The  takeoff  times  are  set  by  the  user  at  the 
beginning  of  the  replication  and  remain  constant  through  the  replication.  The  default 
times  are  0800,  1000,  1200,  and  1400.  The  aircraft  are  released  into  the  next  day’s  cycles 
four  at  a  time.  If  an  aircraft  goes  into  the  prevent  maintenance  process  or  is  held  in 
unscheduled  maintenance  longer  than  a  day,  that  aircraft  would  return  to  the  hold  area 
after  completion  of  these  tasks  and  then  wait  for  the  next  scheduled  takeoff  time.  This 
process  will  be  explained  further  in  the  Hold  section. 

Before  leaving  this  area,  the  aircraft  are  assigned  an  initial  time  until  failure  for 
the  four  simulated  F-16  radar  LRUs  and  time  since  last  phase  inspection.  The  LRU 
failure  time  is  a  random  draw  from  the  exponential  distribution  with  the  mean  set  at  the 
beginning  of  the  replication.  These  failure  times  are  based  on  the  input  analysis  of  the 
REMIS  data.  The  user  does  have  the  option  to  change  these  numbers  on  the  GUI  screen. 
These  times  are  setup  like  a  countdown  until  failure  time.  Once  the  simulation  detects  a 
failure,  described  in  a  later  paragraph,  then  part  of  the  simulated  repair  process  is  to  make 
a  new  random  draw  from  the  exponential  distribution  with  the  same  mean  time  from  the 
beginning  of  the  replication. 

The  random  draw  for  time  since  last  phase  inspection  is  set  up  similar  to  the  draw 
for  failure  except  in  the  model  when  an  aircraft  accumulates  flying  time,  this  time  is 
added  to  the  time  since  last  phase  inspection  that  was  drawn.  This  random  draw  from  the 
uniform  UNIF(0,300)  distribution  gives  an  aircraft  an  initial  time  since  last  phase 
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inspection  so  that  all  the  aircraft  are  not  entering  phase  maintenance  at  the  same  time. 
Also  different  from  the  failure  time,  once  each  aircraft  has  reached  300  hours  the  phase 
inspection  is  started.  Then  once  completed,  their  time  since  last  phase  inspection  is  reset 
to  zero.  The  aircraft  entities  leave  the  create  area  and  are  routed  to  the  mission 
preparation  area  with  the  routing  time  set  to  zero. 

Mission  Preparation 

The  literature  states  that  the  real  world  sortie  generation  cycle  starts  with  the 
landing  of  the  aircraft.  However,  it  was  more  convenient  to  start  the  simulation  with  the 
mission  preparation  area.  As  stated  above,  the  flying  schedule  is  a  users  choice,  the 
default  is  starting  mission  preparation  at  0800,  1000,  1200,  and  1400. 


Figure  9.  Mission  Preparation  Area 
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Figure  9  shows  the  modules  and  flow  of  the  mission  preparation  area.  Since  there 
is  no  time  change  in  the  movement  from  the  previously  described  create  area  or  from  the 
hold  area  described  later,  these  are  the  times  the  mission  preparation  will  start  for  the 
aircraft.  Each  entering  aircraft  passes  through  a  record  module  to  record  a  sortie  being 
scheduled,  next  it  is  assigned  a  starting  time  of  TNOW,  which  is  the  current  simulation 
time  in  Arena®.  This  is  the  global  time  of  the  simulation  and  is  used  various  places  to 
facilitate  calculation  of  failure  times  and  process  times  for  further  output  analysis.  A 
decision  module  detennines  if  an  aircraft  system  other  than  the  radar  has  failed.  This 
check  is  a  simple  decision  based  on  certain  percentage  of  preflight  failures  that  is  set  at 
the  start  of  the  simulation.  This  check  is  done  before  the  fueling  and  weapons  loading 
process  modules  since  an  aircraft  would  checked  for  failure  before  refueling.  If  an 
aircraft  is  selected  to  have  a  preflight  failure  it  has  an  attribute  called  Status  Bit  that  is  set 
to  zero  so  later  decision  modules  know  where  the  entity  came  from.  After  the  assign 
module  the  aircraft  passes  through  a  record  module  to  note  that  a  preflight  failure  has 
occurred  and  then  it  is  routed  to  the  preflight  station  of  unscheduled  maintenance.  The 
unscheduled  maintenance  will  be  described  in  a  later  paragraph.  If  the  aircraft  passes  the 
inspection  then  it  enters  the  refueling  and  weapon  loadings  modules.  These  are  separate 
process  modules  that  seize  the  resource  refueler  and  weapon  loader  and  delay  the  aircraft 
for  a  triangle  distribution  of  TRI(8,10,12)  and  TRI(25,  30,  35)  minutes  respectively.  The 
capacity  of  each  resource  is  eight,  simulated  for  the  24  hour  day.  The  model  has  the 
ability  to  move  a  certain  number  of  aircraft  back  to  mission  preparation  if  they  are  being 
simulated  to  be  an  integrated  combat  turn.  This  means  an  aircraft  lands  and  parks  and 
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immediately  returns  to  the  mission  preparation  area.  Similar  steps  are  conducted  with 
process  modules  for  the  turn  around  area  except  these  times  are  adjusted.  These  times  are 
TRI(4,  5,  6)  and  TRI(10,  15,  20)  for  refueling  and  weapons  loading.  These  times  are 
adjusted  since  it  should  not  require  the  same  length  of  time  for  a  turn  around  mission. 

The  total  preparation  time  before  the  first  scheduled  takeoff,  which  includes  mission 
preparation,  preflight,  taxi  and  takeoff,  is  about  2.5  hours.  Next  the  aircraft  are  routed  to 
the  preflight  area. 

Preflight  Inspection 

The  simulated  aircraft  have  been  refueled  and  weapons  loaded  and  are  ready  for 
the  preflight  inspection  by  the  crewchief.  Figure  10  shows  how  this  process  is  split  into 
two  based  on  if  an  aircraft  is  performing  a  turn  around  flight. 


Figure  10.  Preflight  Area 
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Once  the  aircraft  have  arrived  to  the  preflight  station  they  travel  through  a  process 
module  where  they  seize  a  crewchief  and  are  delayed  via  a  random  draw  from  the 
triangle  distribution  TRI(50,  60,  70).  If  being  routed  from  the  turn  mission  preparation 
area,  they  are  only  delayed  TRI(5,  10,  15).  After  they  are  finished  with  that  process  the 
clock  is  started  on  the  LRUs.  This  is  done  by  setting  the  LRU  failure  start  time  attributes 
for  each  aircraft  equal  to  TNOW.  Later  in  the  simulation  these  attributes  are  compared  to 
the  current  TNOW  and  that  difference  is  used  to  compute  the  time  that  a  LRU  was 
operating.  This  process  decreases  the  time  before  failure  for  the  LRU  and  eventually  will 
trigger  a  failure.  The  aircraft  have  completed  refueling,  weapon  loading  and  preflight 
and  they  are  now  ready  for  taxi  and  takeoff. 

Aircraft  Launch 

Since  the  aircraft  have  been  prepped  for  flight,  they  are  ready  for  engine  start, 
final  system  check,  and  taxi.  Figure  1 1  shows  the  aircraft  launch  area. 


Figure  11.  Aircraft  Launch  Area 
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The  aircraft  enters  a  process  module  where  it  seizes  a  taxiway  and  simulates 
engine  start,  final  systems  check,  and  taxiing  to  the  runway.  There  are  four  taxiways 
resources  available.  The  aircraft  are  delayed  according  to  the  triangle  distribution,  TRI(7, 
10,12).  Next  the  LRU  failure  times  are  updated  for  each  aircraft,  this  is  for  failure 
checking  in  this  area.  Then  the  aircraft  enter  a  decide  area  to  determine  if  they  are  to  be 
routed  for  PHM  checking.  The  PHM  checking  happens  at  four  areas  and  it  will  be 
described  in  later  sections.  If  the  PHM  is  off,  then  the  aircraft  entities  are  routed  to  the 
first  failure  checking  area.  Three  of  these  checking  areas  exist  and  the  only  difference 
between  these  areas  is  the  counting  for  ground  or  air  aborts  and  entity  routing.  The 
failure  checking  compares  all  the  LRU  failures  times  against  what  level  is  set,  usually 
zero.  If  all  failure  levels  are  greater  than  zero  then  the  entity  is  routed  to  the  takeoff  area. 
If  a  failure  is  detected  then  the  entity  is  moved  through  an  assign  area  that  corresponds  to 
the  LRU  that  is  indicating  failure.  Its  LRU  failure  time  is  redrawn  from  the  exponential 
distribution  with  the  same  mean  as  before.  Also  in  this  assign  area,  an  attribute  is 
attached  that  indicates  which  LRU  failed.  This  attribute  will  be  used  for  routing  later  in 
the  unscheduled  maintenance  area.  Next,  a  record  module  records  the  ground  abort. 

Then  the  aircraft  is  routed  to  the  unscheduled  maintenance  area.  If  no  failure  exists,  the 
aircraft  is  routed  to  a  process  module  that  simulates  the  takeoff.  The  aircraft  grabs  the 
only  runway,  if  available,  and  then  is  delayed  by  the  triangle  distribution,  TRI(2,  3,  4). 
There  is  only  one  runway  simulated  so  the  other  aircraft  must  wait  for  the  aircraft  to 
takeoff  or  land  if  it  is  already  being  used.  After  the  process  delay  the  aircraft  is  routed  to 
flying. 
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Flying 

The  aircraft  has  now  been  prepped,  has  taxied  and  taken  off.  Several  checks  and 
assignments  are  made  in  this  area,  these  are  shown  in  Figure  12. 


Figure  12.  Flying  Area 


After  entering  from  the  taxi  and  takeoff  area,  an  assign  area  makes  a  random  draw 
from  the  normal  distribution  with  mean  2  hours  and  standard  deviation  of  0.5  hours. 
These  are  also  notional  numbers.  If  the  draw  is  less  than  15  minutes  then  the  sortie 
duration  is  set  at  15  minutes.  The  sortie  duration  was  built  from  Hill  AFB,  UT  input  data 
described  above.  Also,  like  the  previous  area,  the  failure  time  is  decreased  according  to 
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how  long  they  have  been  operating.  Once  these  assignments  are  complete  then  the 
aircraft  goes  through  a  decision  module  to  detennine  if  the  PHM  is  turned  on.  If  PHM  is 
being  simulated  the  aircraft  is  routed  to  the  PHM  failure  check  area  described  later.  If  the 
PHM  is  off,  the  aircraft  is  routed  to  check  for  an  air  abort.  Since  the  sortie  duration  is 
assigned  a  priori,  a  check  can  be  made  for  an  air  abort.  The  check  is  made  to  see  if  any 
of  the  LRU’s  remaining  life  is  less  than  the  assigned  sortie  duration.  If  less  than  sortie 
duration,  then  a  new  failure  time  is  drawn,  an  air  abort  is  counted,  and  the  aircraft  is 
routed  to  the  unscheduled  maintenance  area.  If  it  passes  this  check,  then  it  is  routed  to 
the  false  alarm  check  station.  The  false  alarm  check  is  similar  to  the  air  abort  and  ground 
abort  check  except  the  percentage  of  false  alarms  are  set  by  the  user  at  the  GUI  and  they 
are  only  used  when  the  PHM  is  on.  A  decision  module  is  set-up  so  that  equal  chance 
exists  for  each  LRU  failure.  If  a  false  alann  occurs  it  is  counted  and  the  aircraft  is  routed 
to  unscheduled  maintenance.  Otherwise,  the  aircraft  is  routed  to  the  flying  process 
module  where  a  flying  resource  is  seized  and  delayed  for  the  sortie  duration.  Sixteen 
resources  are  available  for  flying  since  it  did  not  make  sense  in  this  model  to  restrict  the 
number  of  pilots.  After  the  sortie  process,  the  time  since  last  phase  inspection  is 
increased  by  the  sortie  duration  and  the  number  of  hours  flown  is  increased  by  that  same 
number.  Next,  a  sortie  is  recorded  to  be  compared  to  the  number  of  scheduled  sorties  for 
the  flying  efficiency  calculation.  Then  the  aircraft  are  routed  to  the  landing  area. 

Landing 

Once  the  aircraft  is  routed  to  the  landing  area,  it  enters  a  process  module  where 
the  runway  is  seized,  if  available,  and  delayed  according  to  the  triangle  distribution, 
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TRI(14,  15,  16).  Figure  13  shows  the  station  and  routing  and  one  process  module  for  this 


area. 


Land. 

Ri  rvway  «  wired  and  r«4«as«d  after  a  short  delay 


Figure  13.  Landing  Area 


As  noted  above  there  exists  only  one  runway  resource,  so  a  landing  aircraft  must 
wait  if  another  entity  has  already  seized  the  runway.  After  landing  processing,  the 
aircraft  is  routed  to  the  parking  and  recovery  area. 


Parking  and  Recovery 

The  parking  and  recovery  area  has  two  main  functions  happening  concurrently, 
these  are  shown  in  Figure  14. 
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Figure  14.  Parking  and  Recovery  Area 


First  the  crewchief  resource  is  seized  and  held  while  the  aircraft  is  moved  through 
this  entire  mode.  Next,  the  aircraft  is  checked  to  see  if  it  already  has  been  chosen  for  a 
turn  around  that  day.  If  so,  it  is  sent  along  a  path  that  will  eventually  take  it  to  the  hold 
area.  If  not,  the  aircraft  proceeds  to  another  decision  module  to  see  if  it  will  be  turned 
around  for  another  consecutive  flight.  This  decision  module  makes  it  determination  on 
how  many  to  turn  around  based  on  the  preset  number  by  the  user.  This  is  modeled  this 
way  to  insure  the  same  aircraft  is  not  repeatedly  turned  around.  This  sets  up  two  identical 
paths  that  the  aircraft  can  take  and  they  have  similar  modules  so  only  one  will  be  defined 
here.  The  aircraft  enters  an  assign  module  to  have  the  parking  and  recovery  time 
determined.  This  time  is  a  random  draw  from  the  triangle  distribution  TRI(5,  7,  9).  Two 
minutes  is  subtracted  from  this  time  but  will  be  accounted  for  in  the  next  module.  This  is 
done  to  allow  two  minutes  for  aircraft  parking  and  then  the  clock  is  stopped  on  the  LRUs, 
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simulating  engine  shutdown.  The  remainder  of  the  parking  and  recovery  time  is 
processed,  the  crewchief  is  released  and  then  the  aircraft  proceeds  to  service  and  debrief 
or  mission  preparation  if  the  aircraft  is  being  turned  around  for  another  flight. 


Service  and  Debrief 

Once  the  aircraft  has  entered  the  service  and  debrief  area  it  is  assigned  service  and 
debrief  times  via  an  assign  module.  These  times  are  from  the  triangle  distribution, 
TRI(45,  60,  75)  for  service  and  TRI(10,15,20)  for  debrief.  These  nodes  are  set-up  to  run 
concurrently,  whichever  time  is  greater  is  processed  first  and  then  the  remainder  of  the 
time  is  processed  in  the  second  delay.  Figure  15  shows  the  split  for  the  larger  process 
time. 
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Figure  15.  Service  and  Debrief  Area 


48 


The  current  process  times  do  not  require  the  next  decision  area  since  the 
maximum  debrief  time  is  less  than  the  minimum  service  time,  but  the  logic  is  included  in 
case  there  was  a  change  to  the  times.  This  decision  area  sends  the  aircraft  to  the  path  that 
has  the  largest  process  time.  Depending  on  which  time  is  larger  either  the  crewchief  or 
maintenance  debriefer  is  seized  first.  Once  the  process  is  complete  for  either  the  service 
or  debrief,  that  resource  is  released  and  the  other  resource  is  seized.  After  the  second 
process  is  complete  the  second  resource  is  released  and  then  the  aircraft  is  routed  to  the 
failure  check  station. 

Failure  Checking 

In  this  area  the  aircraft  is  checked  for  any  LRU  failures,  or  like  the  other  checks 
any  LRU  failure  times  less  than  the  preset  level.  Figure  16  shows  a  view  of  this  area. 


Figure  16.  Failure  Checking  Area 
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After  the  aircraft  enters  the  area,  an  assign  module  assigns  the  Status  Bit  to  three 
and  then  the  aircraft  is  routed  through  a  decision  area  to  see  if  PHM  is  on.  If  this  is  a 
PHM  run,  then  the  aircraft  is  routed  to  the  PHM  area  and  does  not  return.  If  PHM  is  not 
on,  the  aircraft  is  routed  to  the  decision  module  to  see  if  any  LRUs  are  below  the  preset 
failure  level,  again  usually  zero.  If  no  failure  is  present  then  the  aircraft  is  routed  to  be 
checked  for  phase  maintenance  being  required.  If  a  failure  exists  the  failure  time  is 
redrawn  from  the  exponential  distribution  and  the  aircraft  is  routed  to  unscheduled 
maintenance. 

Preventative  Maintenance 

After  debrief  or  unscheduled  maintenance  the  aircraft  are  routed  to  preventative 
maintenance.  Figure  17  shows  a  view  of  this  area. 
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Figure  17.  Preventative  Maintenance  Area 
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The  first  thing  that  happens  is  a  check  to  see  if  enough  flight  time  has 
accumulated  to  require  phase  maintenance.  This  phase  is  done  based  on  a  set  number  of 
flight  hours,  for  the  F-16  that  equals  300  hours.  In  the  Flying  area  the  individual  aircraft 
accumulate  flying  hours  based  on  sortie  duration.  The  starting  point  is  determined  by  a 
random  draw  from  the  uniform  distribution  UNIF(0,300).  This  gives  every  aircraft  a 
different  time  that  it  will  enter  phase  and  then  return  to  phase.  This  is  helpful  since  phase 
maintenance  is  scheduled  to  last  from  5  to  8  days  based  on  a  random  draw  from  a  triangle 
distribution.  The  aircraft  are  then  reset  to  zero  phase  time  so  again  different  spacing  is 
maintained.  After  the  check  to  see  if  enough  time  has  accumulated,  the  aircraft  are  either 
routed  to  the  hold  area  or  to  the  phase  maintenance  area.  In  the  phase  maintenance  area, 
the  aircraft  attributes  are  changed  to  reflect  phase  having  been  conducted,  a  record  area 
updates  the  number  of  aircraft  that  have  been  through  phase,  eight  maintenance  specialist 
are  seized  and  delayed  with  the  aircraft  for  the  time  specified  above,  and  then  once 
released  the  time  in  phase  is  recorded.  The  aircraft  are  routed  to  the  hold  area  where  they 
will  reenter  the  model  based  on  the  predetermined  start  times.  Under  the  PHM  scenario 
similar  activities  take  place  but  the  opportunity  for  unscheduled  maintenance  is  available. 
If  the  aircraft  enters  the  PHM  checking  area  with  an  LRU  almost  ready  to  fail  and  phase 
maintenance  within  the  PHM  Level  of  hours  until  phase  maintenance,  then  the  aircraft  is 
routed  to  the  Prevent  PHM  Mx  Station.  At  this  series  of  modules,  the  aircraft  seizes  the 
maintenance  specialists  and  starts  phase  maintenance  but  then  simultaneously  the  aircraft 
is  routed  to  unscheduled  maintenance  to  repair  the  LRU.  After  returning  from 
unscheduled  maintenance,  the  aircraft  phase  maintenance  is  completed  and  then  the 
aircraft  returns  to  the  hold  area.  The  reason  this  was  modeled  this  way  was  another 
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opportunity  to  save  maintenance  time  while  phase  is  being  conducted.  The  real  world 
will  be  monitoring  for  these  conditions  to  exist  so  even  though  the  PHM  does  not  exist 
the  thought  was  that  this  may  be  a  feature  that  it  would  utilize. 


Hold 


The  aircraft  enter  the  hold  station  and  are  first  delayed  by  the  difference  from  the 
first  takeoff  time  to  the  last  plus  one  hour  and  then  are  routed  to  another  hold  module 
until  triggered  by  another  entity.  Figure  18  shows  these  two  holds  and  the  modules  that 
trigger  the  collection  of  some  of  the  output  statistics. 


Figure  18.  Hold  Area 
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These  new  entities  are  created  per  the  predefined  starting  schedule.  The  entities 
are  created,  then  trigger  a  signal  to  the  hold  module  which  will  release  up  to  four  aircraft 
entities.  There  are  an  infinite  number  of  trigger  entities  created.  If  any  aircraft  are  in 
preventative  maintenance  or  unscheduled  maintenance  then  less  than  four  are  released. 
Two  other  entities  that  are  created  in  this  area,  signal  the  collection  of  the  daily  aircraft 
availability  and  number  of  sorties. 

Unscheduled  Maintenance 

This  is  one  of  the  most  complicated  and  most  important  areas  since  it  determines 
the  maintenance  time  and  supply  times  used  in  post  processing  analysis. 
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Figure  19.  Unscheduled  Maintenance  Area 
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Figure  19  shows  the  view  of  the  unscheduled  maintenance  area.  Aircraft  entities 
enter  this  area  from  various  places  in  the  model  if  an  LRU  is  less  than  the  preset  failure 
level  or  in  the  PHM  case  below  the  PHM  preset  level.  They  can  also  enter  this  area  from 
the  preflight  failure  area  of  the  mission  preparation  area.  Entities  entering  from  the 
preflight  failure  first  are  routed  to  an  assign  module  that  sets  an  attribute  equal  to  one  for 
separation  from  the  radar  failures  at  two  points  of  this  area.  Also,  the  other  starting  time 
attribute  is  set  to  TNOW,  and  lastly  a  variable  that  counts  the  number  of  preflight  failures 
is  updated.  This  is  a  duplicate  counting  from  the  mission  preparation  area  but  a  good 
double  check.  As  stated  in  the  assumptions,  preflight  failures  simulate  a  failure  of  a 
system  other  than  radar  but  no  parts  are  replaced.  Next  the  entity  enters  a  decision  area  to 
determine  if  PHM  is  on.  If  PHM  is  on,  troubleshooting  is  bypassed  since  the  PHM  will 
have  already  conducted  this  troubleshooting.  If  not  on,  the  entity  starts  the 
troubleshooting  process  by  seizing  a  maintenance  specialist  and  is  delayed  by  the  triangle 
distribution,  TRI(20,  24,  30).  After  troubleshooting  the  aircraft  enters  another  decision 
area  for  detennining  which  LRU  to  order  from  supply.  Since  this  is  a  preflight  failure,  no 
order  is  required  and  the  aircraft  is  sent  to  another  decision  module  to  determine  if  PHM 
is  on.  If  not  the  aircraft  enters  a  series  of  delays  that  seize  the  maintenance  specialist  at 
each  processing  module.  These  are  operational  check  TRI(15,  20,  25),  signoff 
discrepancy  TRI(5,  10,  15),  and  document  corrective  action  TRI(5,  10,  15).  If  PHM  is 
on,  these  delays  are  again  bypassed  expect  for  document  corrective  action  since  the  ALS 
system  will  be  able  to  complete  these  items  independently  and  present  documentation  to 
the  maintenance  specialist  via  a  handheld  computer  device  for  a  quick  verification  and 
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sign  off.  Next,  the  aircraft  entity  is  split  from  the  radar  failure  entities  and  another  set  of 
assign  modules  record  the  time  the  entity  has  spent  in  the  system  and  it  resets  status  bits. 
Lastly,  the  aircraft  enters  the  last  decision  module  to  determine  if  this  aircraft  came  from 
PHM  preventative  maintenance,  explained  in  the  previous  paragraph. 

If  a  LRU  failure  occurs  the  aircraft  entity  enters  from  a  different  station  and  is 
assigned  key  attributes.  These  attributes  are  the  same  as  before  but  are  key  to  counting 
radar  failures  and  the  amount  of  hours  that  an  aircraft  spends  in  maintenance.  The 
aircraft  enters  the  decision  module  to  see  if  PHM  is  on.  If  not  troubleshooting  is 
conducted  with  the  same  time  as  above.  Next,  the  aircraft  enters  the  decision  module  to 
see  which  LRU  needs  to  removed,  replaced,  and  ordered.  When  the  aircraft  traveled 
through  the  failed  LRU  assign  module  to  reset  the  failure  time,  that  assign  module  also 
assigned  an  attribute  bit  to  one.  So  the  aircraft  is  routed  to  the  appropriate  LRU  area.  All 
these  LRU  areas  are  the  same  so  only  one  will  be  described.  The  aircraft  first  enters  a 
process  module  that  seizes  a  maintenance  specialist.  The  aircraft  is  delayed  by  the 
triangle  distribution,  TRI(45,  60,  70),  and  then  the  aircraft  enters  an  assign  module  that 
reduces  the  supply  variable  by  one  and  the  supply  time  is  assigned  TNOW.  The  aircraft 
then  enters  a  decision  module  that  detennines  if  a  part  needs  to  be  ordered.  This  level  is 
set  at  the  GUI  screen,  the  default  is  to  order  a  part  as  soon  as  one  is  used.  If  a  part  is  to  be 
ordered  another  decision  module  is  entered  to  determine  if  parts  are  available.  If 
available,  the  aircraft  enters  a  signal  module  to  signal  that  a  supply  entity  needs  to  be 
released  from  its  hold  module.  If  parts  are  not  available,  then  the  part  enters  a  similar 
signal  module,  then  stops  at  the  hold  area  waiting  for  the  supply  entity  to  release  it.  After 
the  supply  entity  releases  the  aircraft  is  enters  a  decision  to  see  if  PHM  is  on.  If  it  is  on, 
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then  the  entity  bypasses  the  time  to  waiting  for  the  part  to  be  delivered  from  supply  since 
the  part  would  have  already  been  ordered  by  the  ALS.  If  PHM  is  off,  the  aircraft  enters  a 
delay  module  TRI(0.5,  2,  2.5)  hours  waiting  for  the  part  to  issued  from  supply  and  it  then 
enters  a  process  module  to  be  delayed  for  the  paperwork  to  be  completed,  TRI(5,  10,  15). 
The  aircraft  then  enters  an  assign  module  to  calculate  the  time  waiting  for  supply  and 
then  enters  a  process  module  to  replace  the  LRU.  In  this  module  a  maintenance  specialist 
is  seized  and  the  aircraft  is  delayed  TRI(60,  84,  120).  After,  that  the  aircraft  enters  the 
same  path  as  described  above  for  a  preflight  failure  at  the  operational  check  area.  Also, 
like  described  above  these  entities  are  split  from  the  preflight  entities  and  times  in 
maintenance  are  updated. 

Supply 

If  flagged  for  repair,  the  aircraft  entity  is  moved  to  the  unscheduled  maintenance  area 
where  it  will  eventually  reach  the  supply  ordering  area.  The  aircraft  are  separated  based 
on  which  LRU  flagged  the  repair  and  sent  to  the  respective  repair  and  ordering  cycle. 
There  exists  four  identical  repair  and  ordering  cycles  in  the  model.  Figure  20  shows  one 
of  these  ordering  areas. 
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Figure  20.  Ordering  of  Supplies  in  Unscheduled  Maintenance 


As  discussed  above,  first  the  aircraft  experiences  a  simulated  removal,  and  then 
the  attributes  and  variables  are  adjusted.  Next  a  decide  area  determines  if  a  replacement 
spare  needs  to  be  ordered.  The  subject  matter  experts  suggested  once  a  part  is  used 
another  is  ordered.  Flowever,  the  user  can  set  a  different  level  at  the  beginning  of  the 
replication  if  so  desired.  If  a  part  is  to  be  ordered  another  decision  area  is  set  up  to  make 
sure  enough  parts  are  available.  The  aircraft  is  held  in  a  Hold  area  until  a  part  arrives  or 
if  a  part  is  available  then  the  aircraft  triggers  another  entity  to  start  the  part  ordering 
process.  This  other  entity  represents  a  supply  truck  being  sent  from  the  depot  or 
manufacturer  to  the  base.  Figure  21  shows  the  routing  of  these  entities.  Note  in  this  area 
that  the  calculated  process  times  are  modeled  as  delays  in  the  routing  from  station  to 
station. 
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Figure  21.  Supply  Area 


Eight  entities  are  held  waiting  for  a  signal  that  a  part  is  required.  If  the  aircraft 
initiates  the  signal  then  only  one  truck  leaves  the  depot  and  proceeds  to  the  base.  This 
model  is  not  intended  to  fully  represent  the  depot  to  base  process,  just  simple  delays  are 
set-up  to  allow  for  simulation  of  supply.  The  delay  for  travel  to  the  base  is  a  triangle 
distribution  of  TRI(0.1,  0.3,  0.5)  days.  Once  the  simulated  part  arrives  on  base  the 
attributes  and  variables  are  updated  and  then  a  signal  is  sent  to  release  one  aircraft  from 
the  parts  not  available  hold.  The  delay  for  the  return  to  depot  is  from  the  uniform 
distribution,  UNIF(0.25,  0.5)  days.  The  supply  entity  returns  to  the  hold  area  awaiting 
another  signal  that  a  part  is  needed.  After  being  released  the  aircraft  goes  through  a 
decision  area  to  determine  if  the  PHM  is  being  simulated.  If  no,  then  the  aircraft  must 
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wait  for  the  part  to  moved  from  base  supply  to  the  flight  and  issued.  This  is  done  since 
the  DIS  will  have  already  notified  supply  that  a  part  is  required  and  will  have  it  waiting 
when  the  aircraft  returns  from  the  mission. 

PHM 

The  PHM  area  conducts  checks  the  same  as  the  other  ground  check  and  air  abort 
and  failure  checking  areas.  However,  at  these  modules  the  variable  used  to  check  for 
failure  is  greater  than  zero.  In  other  words,  the  model  consider  a  part  “failed”  if  its 
remaining  life  is  less  than  some  amount  of  time.  Figure  22  shows  the  decision, 
assignment  and  record  modules  that  make  up  this  area. 
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The  default  setting  is  10  hours  but,  the  user  may  change  that  setting.  This  area  is 
designed  to  return  an  aircraft  back  to  the  ground  or  air  abort  stations  if  they  are  not  below 
2  hours  time  remaining.  This  variable  is  also  adjustable  at  the  GUI  screen.  This  is 
modeled  this  way  so  that  a  ground  or  air  abort  does  not  occur  and  this  may  be  the  way 
that  the  real  PHM  is  modeled.  As  stated  in  the  literature  search,  the  PHM  will  be 
equipped  with  sensors  and  reasoners  to  try  and  determine  the  best  time  for  maintenance. 
If  maintenance  can  be  delayed  because  LRU  life  remains  and  it  would  not  be  a  safety 
hazard,  then  operations  may  continue  to  a  more  convenient  time. 


Model  Verification  and  Validation 

The  verification  and  validation  and  was  done  in  two  steps.  The  verification  was 
done  by  the  model  builder.  Each  area  was  scrubbed  for  correct  modeling,  the  animation 
was  observed  on  several  occasions  to  check  proper  aircraft  flow,  and  the  output  was 
compared  against  real  world  data  to  check  accuracy  of  the  simulated  output.  The 
validation  was  conducted  with  six  subject  matter  experts  (SME)  from  AFRL  Human 
Effectiveness  Directorate,  Logistics  Readiness  Branch  (2002).  The  different  model  areas 
were  reviewed  by  the  SMEs  and  each  process  was  evaluated  for  accuracy.  The  suggested 
changes  included  times  for  turning  on  the  clock  for  the  LRUs,  changes  to  process  times 
like  preflight,  service,  phase,  fueling,  and  weapon  loading,  and  the  number  of  sorties  and 
length  of  sorties  were  discussed.  These  suggestions  were  incorporated  into  the  model 
that  was  previous  described. 
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A  final  verification  analysis  was  conducted  by  running  10  replications  for  one 
simulated  year  equal  to  250  hours  50  weeks  times  5  days  per  week.  The  output  was 
scrubbed  for  reasonableness  and  compared  to  the  Hill  sortie  data.  The  outputs  that  were 
analyzed  were  Sorties/day,  MC  rates,  number  of  spares  ordered/number  of  items  repaired, 
spares  levels,  and  flying  effectiveness.  The  results  of  the  final  verification  are  located  in 
Appendix  A. 

Conclusion 

The  model  for  this  research  was  built  to  replicate  the  sortie  generation  process  and 
the  future  ALS  in  order  to  measure  its  effect  on  this  important  process.  This  chapter 
detailed  the  processes  that  make  up  the  model.  Sixteen  aircraft  enter  into  the  simulated 
day  in  groups  of  four,  at  four  distinct  times.  The  aircraft  are  subject  to  numerous 
processes  that  first  prepare  them  for  flight,  simulate  taxi,  takeoff,  flying,  and  landing,  and 
then  they  are  serviced  and  debriefed  all  in  the  means  of  returning  the  aircraft  to  flying 
status.  Returning  the  aircraft  to  flying  status  may  involve  repair  and  replacement  of  a 
certain  radar  LRU  or  as  simple  as  troubleshooting  and  documentation.  The  ALS 
simulates  the  aircraft  being  monitored  continuously  for  anticipated  faults  and  repairs  are 
made  according  to  a  predetermined  level  before  the  fault  occurs.  The  main  focus  of  the 
ALS  has  been  on  the  PHM  and  assumptions  were  made  that  DIS  would  be  fully 
functional.  The  PHM  did  include  false  alarms  that  were  significantly  detrimental  to  the 
sortie  generation  process  since  with  every  false  alarm  a  LRU  is  removed  and  replaced, 
believing  that  the  PHM  is  always  correct  and  has  fault  isolated  to  the  correct  part.  Lastly, 
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the  final  verification  runs  were  extremely  helpful  in  learning  a  great  more  about  the 
model  with  the  review  of  output  that  will  be  similar  to  the  analysis  output. 
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IV.  Analysis  and  Results 


Introduction 

The  previous  chapter  defined  the  simulation  model  that  was  used  for  this  research. 
This  chapter  defines  the  steps  in  setting  up  and  perfonning  our  analysis.  These  steps 
include  detennining  the  appropriate  length  and  number  of  replications  to  produce 
sufficiently  normal  output  data  while  meeting  a  specified  confidence  interval  half  width. 
This  discussion  then  sets  up  the  analysis  of  the  results  from  the  simulation  runs. 


Simulation  Output  Variables 

PACAFI  21-102  lists  the  logistics  perfonnance  terms  that  a  unit  must  report  to 
higher  headquarters.  The  output  of  this  simulation  model  was  structured  after  this 
instruction. 


Table  5.  Logistics  Performance  Measures 


Identifier 

Average 

Half-width 

Minimum 

Maximum 

Possessed  Hrs 

30000 

0 

30000 

30000 

AVG  Possessed  ACFT 

16 

0 

16 

16 

MC  Flours 

24101 

62.176 

23750 

24446 

MC  Rate 

0.80337 

0.00207 

0.79167 

0.8149 

NMCM  Hours 

4151.1 

40.092 

3950.4 

4346.2 

NMCM  Rate 

0.13837 

0.00134 

0.13168 

0.14487 

NMC-S  Hours 

1747.6 

24.893 

1590.9 

1903.7 

NMC-S  Rate 

0.05826 

8.30E-04 

0.05303 

0.06346 

Sorties  Scheduled 

21778 

18.534 

21666 

21881 

Sorties  Flown 

20110 

25.728 

19968 

20237 

Hourly  UTE  Rate 

1280.3 

1.6213 

1272.2 

1289.9 

Sortie  UTE  Rate 

1256.8 

1.608 

1248 

1264.8 

Hours  Flown 

20486 

25.941 

20356 

20639 

Flying  Sch  Eff  Rate 

0.9234 

7.88E-04 

0.91706 

0.92728 

Air  Aborts 

586.4 

9.2633 

554 

641 

Ground  Aborts 

111.3 

3.6937 

94 

130 

Abort  Rate 

0.03572 

5.11E-04 

0.03339 

0.03844 
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Table  5  lists  these  output  variables  and  shows  their  magnitudes  and  associated 
confidence  interval  half  widths  for  our  baseline  scenario  with  thirty  replications.  The 
PACAFI  includes  more  performance  parameters  but  it  felt  was  that  those  shown  in  Table 
5  represent  variables  that  were  important  to  this  research.  In  the  next  paragraph  these 
variables  were  scoped  down  to  key  measures  of  effectiveness  (MOE)  that  would 
represent  an  analysis  of  an  aircraft  equipped  with  an  ALS. 

Key  Measures  of  Effectiveness 

Preliminary  runs  of  the  simulation  model  plus  the  baseline  runs  discussed  in  the 
previous  chapter  helped  to  make  key  decisions  on  the  MOEs  that  were  analyzed.  These 
MOEs  are  Mission  Capable  Rate,  Not-mission  Capable  for  Maintenance  and  Supply,  and 
Flying  Scheduling  Effectiveness.  The  Mission  Capable  and  Not-Mission  Capable  rates 
are  based  on  the  number  of  hours  that  an  aircraft  spent  in  each  of  these  status  categories. 
The  Not  Mission  Capable  hours  for  Maintenance  included  both  the  preflight  failures  and 
the  radar  failures,  while  the  Not  Mission  capable  rate  for  supply  tallied  the  time  an 
aircraft  waited  for  a  specific  LRU.  The  Flying  Scheduling  Effectiveness  rate  is  based  on 
the  number  of  flown  sorties  divided  by  the  total  number  of  scheduled  sorties.  It  was  felt 
that  these  rates  would  offer  the  best  way  to  observe  the  differences  between  the  baseline 
system  and  the  ALS,  and  also  the  differences  between  the  different  ALS  levels  that  were 
set-up. 
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Results 


The  goal  for  the  half  width  variation  for  the  key  MOEs  was  plus  or  minus  2%. 

The  majority  of  the  baseline  runs  were  conducted  at  250  days,  which  is  equivalent  to  50 
weeks  of  work  at  five  days  a  week  with  two  weeks  dropped  for  holidays.  This  run  length 
combined  with  10  replications  provided  a  confidence  interval  half  width  of  less  than  one 
percent,  which  is  well  under  the  goal  that  had  been  set.  However,  at  just  250  days  not 
every  aircraft  was  experiencing  the  phase  inspection  downtime.  Extending  the  run  length 
to  5  years  or  1250  days  allowed  every  aircraft  to  go  through  that  inspection  at  least  once. 
This  replication  length  combined  with  10  replications  provided  confidence  interval  half 
widths  on  the  order  of  0.2  percent,  well  below  the  goal  of  plus  or  minus  2%  half  width. 

The  next  step  was  to  determine  the  number  of  replications  that  produced  data  that 
exhibited  normal  behavior.  Utilizing  the  software  tool  JMP®,  the  Mission  Capable  Rate 
and  Flying  Effectiveness  outputs  distributions  were  plotted.  Two  aspects  of  this 
distribution  were  examined,  first  they  were  fit  to  a  nonnal  distribution  and  second  the 
shape  of  the  distribution  was  observed  from  the  graph.  The  data  from  the  10  replication 
runs  passed  the  Shapiro-Wilk  goodness  of  fit  (GOF)  test  at  the  alpha  level  of  0.05. 
However,  the  plot  of  the  distribution  did  not  appear  to  have  the  normal  bell  shaped  curve, 
so  the  number  of  replications  was  increased  to  20.  Figure  23  shows  the  Mission  Capable 
Rate  distribution  for  20  replications.  Figure  24  show  the  Flying  Effectiveness  for  20 
replications. 
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Figure  23.  Mission  Capable  Rate  Distribution  for  20  Replications 


Figure  24.  Flying  Scheduling  Effectiveness  Rate  Distribution  of  20  Replications 


With  20  replications  the  output  again  passed  the  Shapiro-Wilk  GOF  test  at  the 
0.05  level  but  the  distribution  still  did  not  appear  nonnal,  so  the  replications  were 
increased  to  30.  This  output  again  passed  the  Shapiro-Wilk  GOF  test  and  the  graphed 
distribution  appeared  to  have  the  normal  bell  shape.  Therefore  it  was  decided  to  use  this 
combination  of  30  replications  and  5  years  length.  It  also  should  be  noted  here  that  the 
execution  time  for  each  replication  at  5  years  was  quite  small  (two  minutes),  therefore  the 
increased  run  time  in  going  from  10  to  30  replications  was  minimal.  Figure  25  shows  the 


66 


distribution  for  Mission  Capable  Rate  at  30  replications.  Figure  26  shows  the  distribution 
for  Flying  Effectiveness  at  30  replications. 


Figure  25.  Mission  Capable  Rate  Distribution  of  30  Replications 


Figure  26.  Flying  Scheduling  Effectiveness  Rate  Distribution  of  30  Replications 
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Once  the  number  of  replications  and  replication  length  were  determined,  the 
critical  factors  and  levels  of  those  factors  were  determined.  Examining  the  twenty-two 
(PHM  on  or  off  was  not  considered)  possible  input  variables  to  change,  it  was  detennined 
that  the  PHM  Level  and  False  Alarm  Percentage  were  the  two  factors  that  were  most 
important  to  examine.  The  reasoning  that  leads  to  this  decision  can  be  broken  down  by 
the  groups  of  variables.  First  the  MTBF  numbers  were  fixed  because  examination  of 
improved  reliability  was  not  the  goal  of  this  research.  The  supply  numbers  were  fixed 
since  again  supply  was  not  an  area  of  interest  to  investigate.  Like  the  others,  takeoff 
times  were  not  of  a  major  concern.  Fixing  the  preflight  failure  level  and  the  mean  time 
until  phase,  and  number  of  aircraft  to  turn  helped  to  focus  the  scope  to  the  key  factors. 
Lastly,  the  baseline  runs  showed  that  the  second  PHM  level  variable  had  little  effect  on 
the  MOEs,  so  it  was  fixed.  That  left  the  two  factors  that  were  the  main  concentration  of 
the  research  and  the  runs  were  set  up  to  produce  output  that  would  efficiently  and 
effectively  examine  the  space  of  the  false  alarm  and  PHM  level  factors.  Therefore,  a  3“ 
full  factorial  design  of  experiments  was  used. 

The  number  of  false  alanns  are  based  on  the  number  of  sorties  simulated.  As 
discussed  in  Chapter  3,  the  aircraft  moves  through  a  decision  module  in  the  flying  area 
and  a  false  alarm  occurs  based  on  a  predetermined  percentage.  This  simulates  an  air 
abort  and  a  LRU  being  removed  and  replaced.  This  is  a  worse  case  scenario  since  no 
further  troubleshooting  is  completed  to  verify  if  this  was  a  fault  or  a  false  alann.  This 
was  purposely  simulated  this  way  to  examine  the  worse  case.  Also,  it  was  determined 
that  the  lowest  setting  for  false  alarm  percentage  would  be  one,  meaning  that  false  alarms 
were  always  on.  This  factor  seems  like  it  would  have  the  greatest  impact  on  the 
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simulated  sortie  generation  process  since  several  actions  are  started  once  a  false  alarm  is 
detected.  The  PHM  level  factor  is  based  on  a  time  prior  to  the  failure  of  the  LRU.  The 
times  can  be  translated  as  knowing  when  a  failure  was  about  to  occur  with  a  degree  of 
certainty.  A  low  setting  meant  better  certainty  of  an  impending  failure  while  higher 
meant  not  as  sure. 

The  baseline  runs  showed  a  significant  drop  off  of  key  output  measures  of 
effectiveness  with  false  alarm  rates  over  5%.  Therefore  the  false  alarm  rate  was  limited 
to  5%.  It  was  also  felt  that  with  an  operating  ALS  there  would  always  be  false  alarms,  so 
the  minimum  percentage  was  set  to  one  and  the  center  point  at  three.  The  baseline  runs 
additionally  provided  insight  into  selecting  PHM  levels.  The  Mission  Capable  rates 
dropped  slightly  with  increasing  PHM  level.  This  makes  sense  since  overall  LRU  life  is 
limited,  and  increasing  the  time  before  actual  failure  when  an  LRU  is  removed  from  the 
aircraft  would  decrease  the  overall  life  and  in  turn  the  overall  Mission  Capable  rate. 
However,  this  will  be  a  trade-off  that  the  ALS  designers  will  need  to  study  since  in-flight 
and  unscheduled  maintenance  hurts  flight  operations,  but  removal  and  replacement  too 
early  is  more  costly.  The  approach  that  was  taken  for  this  research  was  that  setting  the 
PHM  level  at  5  hours  to  represent  the  system  was  predicting  failure  to  an  accurate  level, 
while  a  high  of  15  hours  meant  that  the  PHM  was  not  as  accurate  in  predicting  failure. 
This  left  the  center  point  of  10  hours.  Also,  the  minimum  of  5  hours  provided  for  no 
ground  or  air  aborts  Table  6  shows  the  two  factors  and  the  levels  that  were  simulated. 
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Table  6.  Simulation  Factor  Levels 


Low 

Center 

High 

False  Alann  (%) 

1 

3 

5 

PHM  Level  (hours) 

5 

10 

15 

Output  Analysis 

With  the  design  of  experiments  presented,  simulation  runs  were  conducted  and 
then  output  was  analyzed.  First  the  means  of  the  MOEs  for  the  nine  ALS  runs  were 
graphed  for  each  key  MOE  as  shown  in  Figures  27,  29,  3 1,  and  33.  These  graphs  all 
indicate  that  the  various  MOEs  do  not  change  significantly  across  different  PHM  levels, 
but  do  vary  with  false  alann  level.  There  is  more  discussion  on  this  topic  in  the  Analysis 
of  Variance  section  later.  Figures  28,  30,  32,  and  34  shows  the  PHM  level  (5  hours)  that 
produces  the  best  MOE  for  each  false  alarm  level  with  the  mean  for  our  baseline  system 
without  an  ALS.  For  the  Mission  Capable  rate,  the  false  alarm  level  of  5%  does  not  show 
any  improvement  over  the  non  ALS  system. 


MCRate 


PHM  5  PHM  PHM  PHM  5  PHM  PHM  PHM  5  PHM  PHM 

10  15  10  15  10  15 


Figure  27.  Mission  Capable  Rate  ALS  Runs 
70 


MCRate 


Figure  28.  Mission  Capable  Rate  Baseline  and  ALS  Runs 
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Figure  29.  Not  Mission  Capable  Rate  for  Maintenance  ALS  Runs 


71 


NMCM  Rate 


Figure  30.  Not  Mission  Capable  Rate  for  Maintenance  Baseline  and  ALS  Runs 
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Figure  31.  Not  Mission  Capable  Rate  for  Supply  ALS  Runs 
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Figure  32.  Not  Mission  Capable  Rate  for  Supply  Baseline  and  ALS  Runs 
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Figure  33.  Flying  Scheduling  Effectiveness  ALS  Runs 


Figure  34.  Flying  Scheduling  Effectiveness  Baseline  and  ALS  Runs 


The  decrease  in  Mission  Capable  Rate  is  driven  more  by  the  increase  in  Not 
Mission  Capable  for  Maintenance.  Comparing  the  scales  of  the  two  Not  Mission  Capable 
graphs,  the  Maintenance  level  is  twice  that  of  supply.  Also,  the  ALS  with  3%  false  alarm 
percentage  for  Not  Mission  Capable  for  Maintenance  exceeds  the  baseline  level. 


Analysis  of  Variance 

To  compare  the  different  ALS  levels  that  were  simulated  an  Analysis  of  Variance 
(ANOVA)  (Wackerly  et  al;  2002)  was  used.  This  analysis  would  compare  the  different 
effects  of  the  two  variables  that  are  shown  in  Table  6.  Tables  7,  8,  9,  and  10  show  the 
ANOVA  of  each  MOE. 
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Table  7.  ANOVA  of  Mission  Capable  Rate 


Summary  of  Fit 


RSquare 

0.98216 

RSquare  Adj 

0.981959 

Root  Mean  Square  Error 

0.005068 

Mean  of  Response 

0.830592 

Observations 

270 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

3 

0.37618206 

0.125394 

4881.459 

Error 

266 

0.00683296 

0.000026 

Prob  >  F 

C.  Total 

269 

0.38301502 

<.0001 

Lack  Of  Fit 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Max  RSq 

Lack  Of  Fit 

5 

0.00009227 

0.000018 

0.7145 

0.9824 

Pure  Error 

261 

0.00674069 

0.000026 

Prob  >  F 

Total  Error 

266 

0.00683296 

0.6130 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

T  Ratio 

Prob>|t| 

Intercept 

0.8305923 

0.000308 

2692.8 

0.0000 

FA*PHM 

-0.001288 

0.000463 

-2.78 

0.0058 

FA 

-0.045388 

0.000378 

-120.1 

<.0001 

PHM 

-0.005363 

0.000378 

-14.20 

<.0001 

Effect  Tests 


Source 

Nparm 

DF 

Sum  of  Squares 

F  Ratio 

Prob  >  F 

FA*PHM 

1 

1 

0.00019897 

7.7457 

0.0058 

FA 

1 

1 

0.37080544 

14435.07 

<.0001 

PHM 

1 

1 

0.00517765 

201.5607 

<.0001 

Table  8.  ANOVA  of  Flying  Scheduling  Effectiveness 


Summary  of  Fit 


RSquare 

0.987949 

RSquare  Adj 

0.987813 

Root  Mean  Square  Error 

0.001761 

Mean  of  Response 

0.925279 

Observations 

270 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

3 

0.06765676 

0.022552 

7269.155 

Error 

266 

0.00082525 

0.000003 

Prob  >  F 

C.  Total 

269 

0.06848202 

<.0001 

Lack  Of  Fit 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Ma  RSq 

Lack  Of  Fit 

5 

0.00000897 

0.0000018 

0.5736 

0.9881 

Pure  Error 

261 

0.00081628 

0.0000031 

Prob  >  F 

Total  Error 

266 

0.00082525 

0.7202 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.9252793 

0.000107 

8631.8 

0.0000 

FA*PHM 

0.0000592 

0.000161 

0.37 

0.7132 

FA 

-0.019387 

0.000131 

-147.7 

<.0001 

PHM 

0.0001131 

0.000131 

0.86 

0.3897 

Effect  Tests 


Source 

Nparm 

DF 

Sum  of  Squares 

F  Ratio 

Prob  >  F 

FA*PHM 

1 

1 

0.00000042 

0.1354 

0.7132 

FA 

1 

1 

0.06765404 

21806.59 

<.0001 

PHM 

1 

1 

0.00000230 

0.7423 

0.3897 
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Table  9.  ANOVA  of  Not  Mission  Capable  Rate  for  Supply 


Summary  of  Fit 


RSquare 

0.940318 

RSquare  Adj 

0.939644 

Root  Mean  Square  Error 

0.001983 

Mean  of  Response 

0.024024 

Observations 

270 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

3 

0.01647906 

0.005493 

1396.975 

Error 

266 

0.00104593 

0.000004 

Prob  >  F 

C.  Total 

269 

0.01752500 

<.0001 

Lack  Of  Fit 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Max  RSq 

Lack  Of  Fit 

5 

0.00006711 

0.000013 

3.5792 

0.9441 

Pure  Error 

261 

0.00097882 

0.000004 

Prob  >  F 

Total  Error 

266 

0.00104593 

0.0038 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.0240238 

0.000121 

199.07 

<.0001 

FA 

0.0095058 

0.000148 

64.32 

<.0001 

PHM 

0.0010852 

0.000148 

7.34 

<.0001 

FA*PHM 

0.0001336 

0.000181 

0.74 

0.4612 

Effect  Tests 


Source 

Nparm 

DF 

Sum  of  Squares 

F  Ratio 

Prob  >  F 

FA 

1 

1 

0.01626496 

4136.473 

<.0001 

PHM 

1 

1 

0.00021197 

53.9067 

<.0001 

FA*PHM 

1 

1 

0.00000214 

0.5446 

0.4612 

Table  10.  ANOVA  of  Not  Mission  Capable  Rate  for  Maintenance 


Summary  of  Fit 


RSquare 

0.984275 

RSquare  Adj 

0.984098 

Root  Mean  Square  Error 

0.003759 

Mean  of  Response 

0.145384 

Observations 

270 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

3 

0.23520471 

0.078402 

5550.03 

Error 

266 

0.00375760 

0.000014 

Prob  >  F 

C.  Total 

269 

0.23896231 

<.0001 

Lack  Of  Fit 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Max  RSq 

Lack  Of  Fit 

5 

0.00004983 

0.000010 

0.7015 

0.9845 

Pure  Error 

261 

0.00370777 

0.000014 

Prob  >  F 

Total  Error 

266 

0.00375760 

0.6227 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.1453838 

0.000229 

635.60 

0.0000 

FA 

0.0358817 

0.00028 

128.08 

<.0001 

PHM 

0.0042786 

0.00028 

15.27 

<.0001 

FA*PHM 

0.0011543 

0.000343 

3.36 

0.0009 

Effect  Tests 


Source 

Nparm 

DF 

Sum  of  Squares 

F  Ratio 

Prob  >  F 

FA 

1 

1 

0.23174964 

16405.51 

<.0001 

PHM 

1 

1 

0.00329517 

233.2645 

<.0001 

FA*PHM 

1 

1 

0.00015990 

11.3192 

0.0009 
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The  Mission  Capable  Rate  ANOVA  shows  that  there  exists  statistical  significant 
difference  between  the  different  effects  simulated  and  that  all  factors  including  the 
interaction  factor  were  significant  at  the  alpha  level  of  0.05.  The  interaction  factor 
indicates  that  a  linear  model  is  not  appropriate  for  the  model  response  surface,  but  the 
lack  of  fit  p-value  exceeding  an  alpha  of  0.05  indicates  that  second  order  factors  are  not 
needed  to  describe  the  model.  The  Flying  Scheduling  Effectiveness  Rate  ANOVA  is  also 
significant.  However,  it  differs  from  the  Mission  Capable  rate  ANOVA  since  only  the 
false  alarm  factor  was  significant  at  the  alpha  level  of  0.05.  This  was  an  interesting 
discovery  about  the  PHM  level  indicating  that  changes  over  that  factor’s  range  would  not 
affect  the  ability  to  produce  sorties,  if  that  was  the  most  significant  factor  for  a  flying 
wing.  The  ANOVA  for  Not  Mission  Capable  Rate  for  Supply  showed  that  the  effects 
were  significant  and  that  the  two  factors  were  significant  but  the  interaction  was  not. 

Like  the  overall  Mission  Capable  Rate  ANOVA,  the  Not  Mission  Capable  Rate  for 
Maintenance  results  showed  that  each  factor  and  the  interaction  were  clearly  significant. 
Also,  the  Not  Mission  Capable  Rate  for  Supply  MOE  failed  the  lack  of  fit  test. 
Investigations  into  the  higher  order  tenns  showed  that  the  second  order  terms  were 
appropriate  to  fit  this  MOE.  The  results  of  this  analysis  are  not  shown. 

Figure  27  shows  the  plot  of  Mission  Capable  Rate  for  the  three  false  alarm  and 
PHM  Levels.  The  rates  do  not  change  significantly  between  the  PHM  levels  but  across 
the  false  alarm  percentages  there  is  a  significant  drop  off  of  Mission  Capable  rate.  Figure 
28  shows  that  at  five  percent  false  alarms  per  scheduled  sortie  the  ALS  equipped  aircraft 
performs  worse  than  the  baseline  aircraft.  Of  course,  the  false  alarm  is  modeled  as  the 
worst  case  possible  as  discussed  in  the  previous  chapter  with  a  part  removed  and  replaced 
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for  every  false  alarm.  In  addition  to  this  fact  there  is  only  one  LRU  on  base  for  the 
Antenna  and  the  MLPRF,  meaning  that  the  entire  system  is  stressed  by  this  false  alarm 
modeling. 

Conclusion 

This  chapter  exhibits  the  results  of  the  simulation  runs  based  on  a  baseline 
situation  without  an  ALS  and  then  the  ALS  two  factor  analysis  displayed  in  Table  6. 
Figure  27  shows  that  an  ALS  equipped  aircraft  performs  better  than  the  baseline  aircraft 
up  to  a  point  with  increasing  false  alarm  rates.  The  cross  over  for  Mission  Capable  rate 
occurs  after  a  false  alann  rate  of  3%,  while  the  Flying  Scheduling  Effectiveness  was  no 
better  at  the  3%  level.  The  Not  Mission  Capable  for  Maintenance  rate  drives  the  negative 
impact  on  overall  Mission  Capable  rate.  This  can  be  tied  to  the  number  of  parts  being 
removed  and  replaced  with  a  higher  false  alarm  rate.  As  discussed  in  Chapter  3, 
troubleshooting,  operations  check,  and  documentation  can  be  bypassed  under  an  ALS  but 
the  time  to  remove  and  replace  a  LRU  does  not  change.  This  exhibits  the  importance  of 
correctly  identifying  the  correct  LRU  to  remove  and  replace.  The  assumption  is  made 
that  the  PHM  correctly  does  that  except  for  a  false  alarm. 
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V.  Conclusion 


Introduction 

The  previous  four  chapters  introduced  the  research  that  was  undertaken, 
reviewed  the  literature,  defined  the  model  and  all  the  different  areas  established  to 
simulate  the  non-ALS  and  ALS  sortie  generation  process,  and  analyzed  the  output  of  the 
results.  The  research  that  has  been  described  by  these  chapters  investigated  the  impact  of 
an  ALS  on  the  U.S.  Air  Force  aircraft  sortie  generation  process.  This  chapter  will  wrap 
up  this  research,  list  lessons  learned  during  this  process,  chronicle  conclusions,  and 
discuss  future  research  areas.  It  should  be  noted  that  the  specific  values  reported  for  our 
MOEs  are  not  necessarily  representative  of  actual  sortie  generation  capabilities  since  we 
are  only  explicitly  modeling  a  single  aircraft  system.  However,  the  improvements  in  our 
selected  MOEs  with  the  introduction  of  the  ALS,  and  sensitivities  of  these  MOEs  to 
variations  in  the  ALS  parameters,  are  representative  of  the  kinds  of  impact  we  anticipate 
with  a  fully  operational  ALS. 

Autonomic  Logistics  Revisited 

The  Autonomic  Logistics  concept  can  be  summarized  as  processes  and  tools  that 
automate  the  approach  to  the  aircraft  maintenance  process  allowing  improved  sortie 
generation  capability  with  less  logistics  needs  (Henley:  2000).  Under  this  concept 
aircraft  maintenance  tasks  such  as  troubleshooting,  parts  ordering,  and  documentation 
would  be  fully  automated  allowing  maintenance  personnel  to  return  aircraft  to  fully 
mission  capable  status  more  quickly  with  less  effort. 
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Autonomic  Logistics  is  still  an  emerging  technology.  Currently  only  the  US 
Army  and  Navy  HUMS  possess  the  capabilities  that  are  approaching  an  automated  state. 
The  HUMS  is  being  installed  in  operational  helicopters  at  this  time. 

Lessons  Learned 

There  were  numerous  lessons  learned  during  every  portion  of  the  research 
process.  Starting  with  the  input  analysis,  the  time  between  replacement  data  was 
formulated  by  hand  from  output  from  the  REMIS  database.  Tools  are  emerging  that  can 
perform  these  tasks  automatically,  this  may  be  an  area  of  future  research.  The  model 
building  was  extremely  time  consuming,  and  starting  this  process  early  in  the  research 
helped  to  allow  changes  in  the  later  stages  of  design.  Documenting  of  the  current  model 
configuration  and  changes  that  were  made  to  each  newer  version  was  also  very  helpful  in 
the  model  building  process.  Converting  Arena®  output  files  to  text  readable  excel  files 
could  have  been  easier  with  the  utilization  of  the  Arena®  file  read  and  write  module. 

Conclusion 

The  literature  search  indicated  a  great  deal  of  research  into  the  ALS  concept.  It 
also  showed  that  discrete  event  simulation  sortie  generation  models  exist,  however,  only 
one  paper  was  discovered  that  discussed  ALS  simulation  outside  of  the  research 
conducted  at  AFIT  by  Capt  Rebulanan  and  Capt  Malley. 

The  results  of  this  research  showed  that  the  ALS  equipped  aircraft  can  perform 
better  than  a  non- ALS  aircraft  up  to  a  point.  The  analysis  from  Chapter  4  shows  an  8% 
improvement  in  Mission  Capable  rate  from  the  baseline  to  the  best  factor  combination  for 
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ALS  (FA  =  1  /  PHM  Level  =  5).  While  the  Flying  Scheduling  Effectiveness  showed  a 
2%  improvement  at  those  same  factor  levels.  At  the  other  factor  levels  combinations  the 
ALS  performed  marginally  better  or  even  worse  then  the  non-ALS  aircraft  as  shown  in 
the  Figures  28,  30,  32  and  34.  The  false  alarm  affect  on  the  MOEs  was  the  most 
interesting  and  a  little  surprising.  Taking  into  consider  that  the  design  in  this  model  was 
most  likely  the  worst-case  scenario,  (part  removal  with  every  false  alann),  it  still  provides 
an  interesting  example  of  an  area  of  concern  for  the  PHM  designers.  If  the  maintenance 
personnel  remove  and  replace  a  part  that  the  ALS  instructs  them  to,  then  this  situation 
creates  more  RTOK  for  events  for  the  depot  which  in  turn  stresses  the  supply  system. 
CND  and  RTOK  have  existed  for  many  years  and  the  rates  do  not  seem  to  be  decreasing. 
Any  new  logistics  concept  should  strive  to  decrease  these  rates. 

Recommendations  for  Further  Study 

A  great  deal  of  opportunities  exist  to  make  significant  improvements  of  this  sortie 
generation  model,  mostly  in  the  PHM  area.  The  fact  that  a  working  system  still  does  not 
exist,  could  lead  to  investigations  into  similar  system  designs  like  the  HUMS  or 
interviews  with  the  Joint  Strike  Fighter  SPO  on  preliminary  designs,  trying  to  determine 
their  direction  so  far  and  future  thoughts  of  what  a  system  may  resemble.  Studying  the 
false  alarms  of  previous  systems  and  the  actions  taken  by  maintenance  personnel  could 
be  further  investigated.  How  the  maintenance  personnel  react  to  false  alarms,  at  what 
point  would  they  start  ignoring  the  system  or  how  could  improvements  be  made  to  gain 
initial  confidence  in  the  system.  Research  could  be  conducted  on  the  process  after  a 
false  alarm  occurs.  Issues  could  be  examined  such  as  do  you  perform  a  remove  and 
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replace  of  the  LRU  with  every  false  alarm.  Also  should  this  remove  and  replace  action 
be  based  on  the  percentage  of  sorties  as  it  was  modeled  or  maybe  a  percentage  of 
operating  time.  Currently  which  LRU  that  is  removed  and  replaced  is  based  on  an  equal 
chance,  maybe  a  better  basis  may  be  on  the  proportion  of  supply  on  base.  Another 
research  avenue  may  be  into  the  PHM  assumptions  such  as  no  troubleshooting  and  no 
waiting  for  base  supply.  Also,  every  DIS  message  was  assumed  to  be  received  by  supply. 
Modeling  could  be  added  that  simulates  dropped  messages,  slow  message  traffic,  or 
downtimes.  Other  issues  about  the  timing  of  DIS  messaging  could  be  investigated,  when 
is  the  information  required,  at  landing  or  earlier  while  still  in  the  landing  pattern. 

One  of  the  original  goals  of  this  research  was  to  build  the  model  to  allow  easy 
transition  to  an  object-oriented  model  and  provisions  were  made  during  the  model 
building  to  allow  that  transition.  The  entities  in  the  model  were  treated  like  objects  and 
were  assigned  attributes.  This  could  lead  to  possible  further  research  using  the  JAVA® 
Silk®  simulation  package. 
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Appendix  A.  Baseline  Runs 


Rates 

Baseline 

PHM2/5/0 

PHM2/5/2 

PHM2/5/5 

PHM2/10/0 

PHM2/20/2 

MC 

0.8001 

0.8584 

0.8584 

0.8560 

0.8462 

0.8361 

NMCM 

0.1407 

0.1230 

0.1234 

0.1251 

0.1310 

0.1413 

NMCS 

0.0592 

0.0186 

0.0182 

0.0189 

0.0228 

0.0026 

FSE 

0.9234 

0.9368 

0.9346 

0.9234 

0.9359 

0.9359 

Rates 

PHM2/20/5 

PHM2/10/2 

PHM2/10/5 

PHM2/20/0 

PHM5/5/5 

MC 

0.8381 

0.8518 

0.8480 

0.8382 

0.7877 

NMCM 

0.1375 

0.1276 

0.1311 

0.1380 

0.1794 

NMCS 

0.0244 

0.0206 

0.0209 

0.0238 

0.0329 

FSE 

0.9342 

0.9371 

0.9325 

0.9345 

0.8916 

Rates 

PHM5/10/5 

PHM5/20/5 

PHM10/5/5 

PHM10/10/5 

PHM  10/20/5 

MC 

0.7787 

0.7619 

0.6730 

0.6513 

0.6273 

NMCM 

0.1847 

0.1967 

0.2595 

0.2752 

0.2941 

NMCS 

0.0365 

0.0415 

0.0675 

0.0735 

0.0787 

FSE 

0.9042 

0.9024 

0.8482 

0.8548 

0.8554 
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Mission  Capable  Rate 
ANOVA  Assumptions 
Constant  Variance 


Appendix  B.  ANOVA  Assumptions 


SSR 

#  X  cols 

SSE 

n 

Breusch-Pagan 

Chi-square 

3.57E-09 

3 

0.006741 

270 

1.907658683 

0.992845 

Independence 

Durbin-Watson 


Durbin-Watson 

Number  of  Obs. 

Autocorrelation 

Prob<DW 

2.0625877 

270 

-0.0323 

0.6539 

Influential  Data  Points 
Cook’s  D 


Studentized  Residuals 


Fitted  Normal 
Parameter  Estimates 


Type 

Parameter 

Estimate 

Lower  95% 

Upper  95% 

Location 

Mu 

0.000052 

-0.119960 

0.120063 

Dispersion 

Sigma 

1.001610 

0.923652 

1 .094053 

Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test _ 


W 

Prob<W 

0.984409 

0.5824 
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Flying  Scheduling  Effectiveness 
ANOVA  Assumptions 
Constant  Variance 


SSR 

#  X  cols 

SSE 

n 

Breusch-Paqan 

Chi-square 

2.12E-10 

3 

0.000816 

270 

7.717642765 

0.5628324 

Independence 

Durbin-Watson 
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Autocorrelation 
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Studentized  Residuals 


Fitted  Normal 
Parameter  Estimates 


Type 

Parameter 

Estimate 

Lower  95% 

Upper  95% 

Location 

Mu 

-0.00019 

-0.120179 

0.119803 

Dispersion 

Sigma 

1.00144 

0.923491 

1.093862 

Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test _ 


W 

Prob<W 

0.983523 

0.5048 
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NMCM 

ANOVA  Assumptions 
Constant  Variance 


SSR 

#  X  cols 

SSE 

n 

Breusch-Pagan 

Chi-square 

1.86E-09 

3 

0.003758 

270 

3.20308814 

0.9556957 

Independence 

Durbin-Watson 


Durbin-Watson 

Number  of  Obs. 

Autocorrelation 

Prob<DW 

1.9580986 

270 
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0.3214 

Influential  Data  Point 
Cook’s  D 
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Type 

Parameter 

Estimate 

Lower  95% 

Upper  95% 

Location 

Mu 

-0.00028 

-0.120286 

0.119728 

Dispersion 

Sigma 

1.00157 

0.923617 

1.094011 

Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test _ 


W 

Prob<W 

0.987636 

0.8304 
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NMCS 


ANOVA  Assumptions 
Constant  Variance 


SSR 

#  X  cols 

SSE 

n 

Breusch-Pagan 

Chi-square 

5.79E-10 

3 

0.001046 

270 

12.85914476 

0.1690841 

Independence 

Durbin-Watson 


Durbin-Watson 

Number  of  Obs. 

Autocorrelation 

Prob<DW 

2.115013 

270 

-0.0606 

0.7960 

Influential  Data  Points 
Cooks  D 
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Fitted  Normal 
Parameter  Estimates 


Type 

Parameter 

Estimate 

Lower  95% 

Upper  95% 

Location 

Mu 

0.000397 

-0.119641 

0.120434 

Dispersion 

Sigma 

1.001825 

0.923850 

1 .094287 

Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test _ 


W 

Prob<W 

0.993117 

0.9918 
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